Okay, it looks like I am live. And hey guys, this is Dr. Nancy Lee. If you see me live on YouTube and LinkedIn, please come on the chat, say hi, what's your name, where you're from, and what's your new year's resolution. Come on the chat, this is Dr. Nancy Lee, we're starting our free coaching and especially free resume review in one minute. You see me live just say hi in the chat and let me play the countdown music. So we're starting one minute, great start. Can you guys hear the countdown music? Come on the chat if you can hear the countdown music or not. I think you should be able to hear the countdown. Sid, you can hear the countdown music? It should be a microphone. Okay, hey guys. It should be able to hear the countdown music now because it's a microphone. No, go back in. Awesome. Hey guys, this is Dr. Nancy Lee. I can see some of you guys are seeing me live and come on the chat. Hello, hello. I see Sid, how are you? All right. So quick introduction, people join us, please come on the chat. Quick introduction, my name is Dr. Nancy Lee and I am a management coach and also director of product and featured in Forbes. I have helped 100 people learn their dream PM job offer in fan companies and unicorn startup and continue to get promoted as a product leader. For lots of people joining us, please continue introduce yourself, what's your name, where you're from and what's your goal for 2023 and come on the chat. I'd love to hear this. Let me see who else is here. I should be able to see everyone's chat. I see the chat directly coming in from YouTube. Hi, Sid, how's it going? You should be able to see me live on LinkedIn as well. Awesome. Awesome. Yes, directly come on the chat and say hi in the chat. Cool. All right. So let's do this. All right. So today is a resume workshop. For people who want to do live resume review, please email me at nancy.nancy.com. Let me put it in the chat so you guys can see it. Nancy Lee. You should be able to see my message. If you want to do live resume review, email me right now and pull you up to the live resume review. If you have any questions, feel free to come on the chat. Annie said, can I become a part of PM3 out of college? Absolutely, yes. Yes, we have lots of people. Became a part of manager right out of college. Actually, for example, we have like Genesis, student Genesis. She studied math, math major, and then she became an APM in the medical tech company. We also have like who said another student, Rachel. Rachel is also a student fresh out of school. She joined Microsoft. Amy also joined Microsoft as a part manager. Yes, you can absolutely become a part of manager fresh out of college. Yes. Cool. All right. So feel free to put your questions in the chat. I should be able to see all the chat right here. Awesome. So I had everything. Now today we do live resume workshop. Let me share my screen. Then we get started. So I show. Okay. So today we do live resume workshop. I want everybody to get ready. Have your resume ready? So do you have your resume ready? If so, feel free to email me at nancy.nancy.com. It's in the chat. Okay. And then you can email me and then put you up to see live resume review and also tell you why you wasn't selected for an interview, what you need to do. Okay. Cool. Specifically, let's go through today's talk. I do want to share with you guys some inspiring stories. This happens every single week that we really want to go. It's really share with you guys regarding do not push yourself on the limiting beliefs. You maybe say, oh, I don't have this. I don't have that. Lots of people have the same challenges, even bigger challenges than you, but they are able to become a part of manager. I want you to use those as inspiration so that you can make it happen as well. Now, specifically, this is one of the students, Addie is crazy. So himself, majority of the experience was in China. Okay. So he worked in China in total for seven years. And he also was entrepreneur for three years right before he became a part of manager and no part management experience. But he was able to become a senior part manager and master card. He also went to several final round interview with TikTok, all the famous companies. What I want to tell you is regarding how you can use the same methodology and do not feel like you don't have it. Actually, lots of people can't make it happen at so many challenges and you can make it happen as well. Okay. So now, specifically, here's what Addie is doing. What his background was that his biggest challenge is he doesn't have like direct management experience. The second challenge that the majority of his work experience is in China. We do have lots of students in India, in Canada, like London, Australia, all over the world in Thailand, Singapore, Philippines, like every country say lots of countries out there. Of course, in China as well. Lots of them, they have their own unique challenges. And for example, to hunker Addie challenge, here's how we did it. Because when he was an entrepreneur, even he had no product management experience, but as an entrepreneur, he did something very similar to product management. That's a straight transition from entrepreneur to product management. Very straightforward. And you should do the same as well. Okay. So do some like startups or even join the startup on the site so that you can build a product portfolio with each product portfolio inside a product manager accelerator program, you can check out more, go to drnc.com. Okay. And then the second thing is regarding work experience in China, not much experience in the US. We just need to repackage his experience, right? So in China, the methodology, how things do things in China is different from the US, but what kind of impact it worked, had created in China, and what kind of experience he had that can be transferable to the employers in the US? All those are important questions he need to present and sell himself to the interview. The interview definitely says, oh, everything in China is not relevant, but you need a battle car. You should tell them, well, here is the relevancy, here is the impact made. And some of my clients, even if I was in China, some clients in the US clients, so many different battle cars, you need to pull off to tell them how you can improve their business bottom line, help them to grow and build a better product. Okay. So that's how Eddie, the CNPM job offer for muscle car. I want you guys to use the same inspiration as well. I believe our royal follower and I was, what's your name, Paul? And Annie, our royal friend Annie, and just as a question, we're going, oh, I don't have experience and fresh out of college. Can I become him? Yes, you can. Okay. We have lots of people with very similar background like you. What I want to tell you is that do not give up. Okay. So you got those experience and you can do all of this. It's more about do you use the right methodology? That's why you are receiving the free coaching from me and help you to grow. We have hundreds of free trainings on YouTube. You should start watching all of them and make sure to like, comment and subscribe. And now also remind everybody, whoever is going into interviews right now, please go to the 50 find interview questions database and go there to download your mock interview questions because you're going to be asked the same kind of interview questions in upcoming interviews. Okay. So which is you can go to this website, docnancy.com slash 50 fan to download all your interview questions. And so whoever wants to join that for free to get free coaching from zoom, you can send email to info, PM, accelerator, diet. Oh, if you want to join me live right now, you can directly email me and Nancy at docnancy.com and just join me live and you're ready to directly receive free coaching from us. Quick reminder, anybody who can receive free coaching could get prioritized or people who are most likes, comments and shares of our YouTube channel and also whoever donated to any of the non-profit. If you already go to my YouTube channel right now, we publish the latest video where we're doing one one donation matches up to $20,000 to help girls from sex trafficking. Okay, we are big believer for our nonprofit mission in the past couple of years. I have donated over $20,000 to the educational non-profit. So this year we're very proud to add Marisi, which is the organization in India and helping girls actually like rescuing girls from sex trafficking. And so $500 can save a girl from sex slavery. And I want all of you guys to help me to share the videos, even if you do not have the money to donate right now or you don't have the budget at this moment, but it's important to let the world out. Okay, other people who want to help, maybe volunteer, maybe they want to do donation, many different ways. Maybe this is sharing. Sharing is loving. Okay, sharing the news and let people know about the craziest crime ever is one of the ways for you to contribute back to the communities. Okay, so let me share with you regarding our, the sex trafficking is crazy. Okay, I am big believer of giving back and helping others. I really need to leverage this opportunity to let you guys know. So go here. It's crazy. Each time I talk about this topic, I cry. I'm probably going to cry on my YouTube live right now. It's crazy. Okay, now let's talk about sex trafficking. This one. So literally I cry like nuts. So let's watch this video.  I just want you guys to share it. Okay, let people know about the mission. You don't need to donate. You don't have the money or you don't want to right now. Don't make sure you share with us. Share, comment and like and subscribe. Okay, it's very important. Yes. Okay, everyone needs to go to this video right now. So let me just put it in the chat. This is one of my biggest mission. It's very sad. Each time I talk about this. Yes, it's crazy. I can't even increase my voice talking about this topic. Cool. All right. So now let's stop sharing and continue with our product management topic, which you guys made me very sad talking about this topic. It's crazy. Okay, so now let's go here. Go back to my deck. Is this my deck here? I think this is my deck. Okay, this is the deck here. Yes, you guys are going to receive free coaching from me, or people have the most shares, likes and comments, subscribe. It's going to receive free coaching from me. This is a very sad topic. Whenever I talk about it, I can't raise my voice because I'm about to cry. Okay, now quick announcement for people. We're going into live resume review right away. Okay, so put your questions in the chat. Let's continue. Okay. Also a quick reminder for people who are interested in product management, accept the program. So it's a coaching program where I help people plan their dream PM job offer in fan companies. You can start up and you can come to my website at darknancy.com slash masterclass. And this masterclass is free if you're interested in PM Accelerator, which is our end to end coaching program. You should stay until the end of the masterclass. But this masterclass also give you lots of value regarding top strategies you need to use to plan and keep your dream program in job during hiring freeze. Okay, so go here, darknancy.com slash masterclass. Continue. I also encourage everybody to take a screenshot right now. Whoever helps you to grow, you need to find the mock interview partners for your upcoming interviews. Communities are important and they are the ones who can help you push you to next level because they support emotional support and resources within the group. Okay. Scan the bar code right now and take a screenshot right now and join either WeChat group or LinkedIn group to find your mock interview partners. All right. And I also want to celebrate regarding our offers. As I said, there are still opportunities out there. I want you guys to understand other people made it happen. You can make it happen as well. We just need to really try to push through your goals, okay, and using the right methodology for example, during Thanksgiving, we have student learning offers from Microsoft service now and now the senior PM offer from Tesla, senior PM offer from Massacars and another offer from Microsoft and like another PM internship offer from Dobli and other interview with like Lenovo, TikTok or eBay and Sharpie, all the famous companies. So you can make it happen as well. Okay. So you need to push hard towards your dream. Life is too short. It's horrible not to work hard towards it. Okay. So now let me answer pre-summit questions. This one question, which is not here. Actually, let me answer the question submit on YouTube and LinkedIn first, whatever is live with us. We answer live questions here. Let me go to live questions once back. Cool. Where is live questions? Okay. So let me go here. Feel free to put questions in the chat. I should be able to see all the questions. By the way, I should be able to see all the questions. Yeah. And these, so your questions, can I become Prime Manager straight out of college? Yes, you can. Let me show you how. Okay. Absolutely. Yes. This happened many times, including like people joining, like straight out of college, joining fan companies as well. Absolutely. Yes. Let me show you how. Let me show you my screen. I'm so very sad about the sex trafficking stuff. And so my mood is low right now. I don't need to teach you guys. This is very sad. Okay. Go to my YouTube channel. Yes. Make sure to share this, share, like and comment. Okay. It's very important mission. Now let's come here. Specifically, if you have no PM experience, fresh out of college and you need to watch this video. Let's watch how to become Prime Manager with no experience. I have a playlist you need to watch right here. So fresh out of college, watch this one. This one. Okay. We have lots of people just like Genesis and you can do it as well. Okay. So basically, she spoke a lot regarding how exactly she learned the APM job offer was in two months and why you should do it as well. Okay. So for her strategies here, as I said, we also have people 90 job offering Microsoft as other places. We'll try to bring them all of them to our YouTube channel and share all the secret behind things. And the faster access regarding how to break into those big tech companies. We also have monthly alumni panel. Usually, we invite some VIP guests to our monthly alumni panel. We should follow our Instagram where we make announcement and you can comment and also request the VIP access. Okay. Follow Doc NCD on my Instagram and you can see some posts out there. Oh, let me show you. Instagram here. Okay.  For example, where is, what's that? Oh, right here. She's right here. So for example, this is our December alumni panel. You can comment in the chat. So to bring opportunity to get invited to our VIP guest. Okay. So this is our student Priyanka. She joined PayPal after she was laid off and she's also on H1B. Okay. She talked about the unique strategies she used to quickly landed seeing a PM offer within two months after laid off. And she still needs like H1B sponsorship since they're getting really, really challenging at the time. Yes. Follow our Instagram. You can comment in the description and we know opportunities to get invited to our alumni panel. Okay. Cool. All right. Now let's go back to how to watch this video. And for fresh out of college, you also need to watch this one. So the skills you need are very critical. Okay. Like six essential skills to become a manager fast. And so study this and six skills, even if you're fresh out of college, you master those skills, you can become a product manager as well for sure. Okay. Now, let me go back to the pre-submitted questions. Feel free to put your questions in the chat. Okay. I'm able to read all of them. Let's also talk about the resume. Anyone want to do resume review? I'm going to review some live resumes, discuss all of this with you guys. What kind of side project to do that can be written on resume? Very good question. What kind of side project to do? And here's how you can do it. This is sit. Sit ask question. What kind of side project? There are, let's go here. I have free training for you as well. For the side project, there are 13 different kind of side projects you can do. Come here. Oops. I shared the wrong thing. This is my backend. Restream software. Let me find the right tab to share. I believe this one is this one. Okay. So right here. Different kind of project you should do. Come here. What my most favorite popular upload and this one, product portfolio. Okay. So in this product portfolio video, I told you how would you be able to use product portfolio to succeed in your resume? To put on your resume and resume. Okay. So lots of things can be put on your resume. And now what is the project? And there is a place you can download 13 different kind of project to add value on your product manager interview. Okay. So go here. You can download a 13 different kind of project. Okay. Go here. Have everything you need. Very important. Now, specifically for all the project, there are many different ways to do project. You can do digital transformation project. You can help start up to launch product. You can also do GTM go-to-market strategy. You can work with developers and help them to create product together. The many different product project you can do so you can put on your resume. Certain different things. And go to the video I just shared with you. And let me put in the chat so you have it as well. Make sure to give the video a like. They already have 678 likes. Make sure you guys do it as well. So let me, let me put in the chat so you guys can go there and watch all the in-depth training. And what's that? Right here. But let me give you the top three project you should do right now. The most effective project you can, you can work on as a side project. It's a digital transformation project, which is my favorite one, which is traditionally there are lots of service industry. They want to go online and you help them to create certain kind of product to do automation to help them to get access to their customers. And also it's kind of like a project is something very favorite because you already have some kind of business you can talk about. You already have some potential customers. You just, you mainly introducing the digital components to help the company to become better on getting access to the customers faster. The number one, number two thing you can do for the project is you immediately start looking to any kind of startups. Okay. So those, which is a GTM go to market strategy for startups. Lots of startups right now, they have some product that really need lots of help to drive customers. So any kind of customer engagement, and you can, you can help startups to get is something you need to work on right away.  So number three is also my personal favorite. Number three is actually you start to create your own product if you want to. Okay. You can, you can do anything. You can create your own product. It's very easy. Any kind of problem you want to solve today and you can interview customers. You figure out how to solve it. You build your MEP. You can do many different, different ways to create your own product, and those three is going to help you significantly. Of course, the video I share with you has 13 different projects you can work on. You can use, you follow the same strategy, and you can just get it without a spot. Okay. Any of the 13 is going to work out. And let me see another question. How to transition from freelancing to product management? Now, yeah, asking me this question. Great. How to transition from freelancing to product management? This is a really good question. Now, freelancing to product management is the following. Number one, when you are doing freelancing work, you probably actually did lots of product management work, which is you already talk to customers on a certain needs and desire, you interview your customers. So the best way for you to transition is actually try to learn product management knowledge upfront, because in freelancing, you probably did a little bit of everything, but you don't know how to do it the right way. How the tech companies, the big tech companies looking for when you actually become a full-time product manager. So first of all, learn what is product management. Now, number two is that when you read package, you will experience doing freelancing. You need to write everything in the product management style on your resume. Okay. Let me show you how to put it on your resume. We should go back to what resume workshop we have right here. Is this the right video? Yes. Okay. So go to portfolio, resume keywords. Let's talk about resume keywords first. Those are the mass wash videos. All of you guys watch this video. You follow our strategies. You're able to learn interviews. Okay. Immediately, increase your success significantly. Okay. Very critical right here. In this video, we are talking about different keywords, which means that when you are a freelancer, it's likely you did customer interview. You wrote some requirements. So talk about different kind of like five different requirements you need to work on. So study all of those. Okay. And then you can put those keywords on your resume and write as an example like how you did it right now. Okay. So study this very carefully.  Resume tips with no PM experience. Once you come here, where I gave you the resume templates, go here in the description. You click here, PM resume template. You can download the template on your own. Okay. This template has been used by 7,000 product managers to land a dream PM job. You should use the same template as well. Okay. So now for your freelancer experience, how everyone put on your resume is that please make sure you talk about your achievement. It's not about what you did. It's about your impact. You have done something and here is your impact. You have done for your employers. Okay. Immediately study your impact. And then you put in two sessions of your resume, which I put it right here. For example, let's pause this video. After you put your impact, you also talk about your unique product management related function as a freelancer. For example, you can interview customers, study market trends to understand the needs and convert them into action. Action mode, engineering, facing user stories, priority user stories and requirements, all of this. All of this is very relevant to product management. You properly did it. EDM did it in the right way to be frank because of freelancer. Okay. So therefore you learn it first and then follow this free training. You can definitely land your job offer as well for sure. Okay. So now what's one I share with you? Oh, how to gain PM experience? Go to my free training playlist. I have like hundreds. Oh, how many videos? Let me see how many videos I have. Do you show the numbers? I don't know. Do you show a number? Make sure to like, share and subscribe and comment. Like, comment, subscribe. I'll make sure to like this video as well. Make sure to hit the like button if you like the free coaching I just gave you. Now, when you learn product management, it's important to understand roles and responsibilities, different kind of top 10 jobs. So the different kind of top 10 jobs need to pay attention to based on your specific experience. And make sure to study what does product manager do in real life. And in that case, you study the differences between product managers. Okay. And how to become like top product manager, the good product manager, bad product manager and study those like training. So you understand what does product manager do. Okay. So those are videos right here and many different things. Of course, if you're interested in in-depth training with me and my team, you can go to this website, docnancy.com to learn more where we discuss in-depth regarding how you're able to join me and getting trained with me and senior PM from Apple and Amazon and other PMs out there. Awesome. So let me see any live questions. If not, we go to pre-summit questions. So let me go to live questions right now. Let me find a browser. Cool. Next question. Ruby said, can you speak a little about transitioning from sales product management? And I haven't tend to add in my resume for PM focus roles, but would appreciate any insight. Okay. Great. So first of all, Ruby, yes, sales into PM, that happened in the past as well. So let me show you my screen, share you my screen. Almost all possible background will have helped people to achieve to become product manager. Almost all possible questions. We have a lot of free training for you and paid training as well. So let me show you where to get the sales. First of all, sales to PM. Let me show you. First of all, go to... First of all, I want to show you guys regarding this can be done. Okay. So go to docnancy.com. Go to testimonial right now. Let me find you who's from sales background. This person. One sec. We have so many. So let me find. Okay. So he came from sales background. He came from insurance industry. Okay. Seven years in the insurance industry. His background is bachelor of business management. He does sales and sales and customer support for the insurance company. Okay. Then he learned two PM job offer. One of those offered PM at JP Morgan Chase. He had been working there for one year already. That's Alex Hangzhou. Yes. You can definitely become product manager with sales background as well. Now let me share with you regarding how come here you are. Now regarding transitioning and how and then we talk about any insight regarding how to put on your resume. Okay. So number one thing is that when we talk about how to become a PM is very critical. You need to understand what does PM do. Okay. And what the day to day look like. Number one thing is you come... I have a video about this. Come here. Go to this playlist. No PM experience. Transition to PM. No experience with a no experience playlist. Great. Okay. Actually only 9% of product manager out there. The first job title is product manager. 91% of them transition from outside product management space. 91%.  Which is you are the minority. You are the majority. Whoever had already had a PM title, the first job, the PM, they are minorities. So you are majority. So you should be having majority of people like you transitioning to product management. Yes, you should and you can. All right. So first of all, let's go here. Where is... Study this one. What does product manager do in Google, Facebook, TikTok and different kind of companies and what do they do? In this video, I cover about different kind of roles and responsibilities you need to cover. You need to do in different companies. And then immediately you will start to make connections between what you did before such as sales, how much sales you are in product management. Extremely relevant because you do everything all the way to talking to customer, understanding needs and desire. You communicate the customer's needs and desire to the engineering team. You probably write some in the past. Hopefully you wrote some requirements and helping your engineering team to understand what customer wants. And this is also very relevant. Okay. If you haven't written requirement, write it today. Okay. Any insight from customers you write it today. Of course, you do need to know how to write it correctly. You cannot just say, I write it and just say not good. People, anyone already have PM experience.  And then it's like you cannot end a job. Okay. So it's very, very important. You immediately use the real product management knowledge to improve it. Okay. But this is in a high level. How you do the transition. Now regarding resume. So lots of your experience, you did, let's say, sales experience on your resume. You just need to convert your sales experience resume into product management resume. For example, you talk to customer, you sell product to them. You convert talking to customers into your contact voice customer interview. Okay. And then you sell product to them. You can talk about you discuss product feature, discuss and recommend product feature to customers to help them to solve problems. Everything put on your resume should be very relevant to product management.  Okay. You are the majority. You are the majority. So majority of you guys came from as a background. You just need to show them you have transferable skills have done something very similar. Just say not through traditional, like the first title as product managers, you always break into product management. You are the 91%. Okay. So yes, you can do it. And of course, you want to get in-depth training where it's through hands-on training from me and senior PM from Apple and other fan PMs and go to darknancy.com to learn more regarding if you're qualified for the program and check out our free YouTube video as well. We have lots of free training out there. Make sure to like comment and share and subscribe. Now let me do this. Oh, I forgot to put this video in the chat. So you guys must watch it. Make sure to like it. Let me do it here. Let me one sec. One sec. Where is the chat? Oh, I have lots of comment coming in. Ruby, thank you. It's very helpful. Thank you. Thank you. Thank you for your question. Okay. So one sec. From Sales background. Okay. We finished Ruby's question. Sumi said, how much tech knowledge is required if someone from NumTek background because I see a lot of company looking for CS degree or good technology understanding? Okay, great. Very good question, Sumi. Depends on the role. Depends on company. If you join Google, yeah, lots of tech knowledge. Google makes it really difficult. Okay. They have software engineer interviewing you, even if recently they announced they're not going to have software engineer interviewing you, but people who currently were already program managers, they joined Google, they're all extremely technical. They were going to ask you technical questions for sure. Even if you're not taking or you joined Google, you joined Google by luck, you're not technical, but everyone else basically will look down on you. You don't know technical stuff. You don't know how to run a server on the cloud. People look down on you in Google and it's very hard for you to keep your job. You easily get fired in Google. Okay. So for those kind of companies, yes, you really need very deep tech knowledge. Now, I recommend you look outside of Google. For example, Meta, right now Meta is not hiring. They only have 28 openings right now. They're cutting off 13% of employees. We don't join Meta, but use that as an example or TikTok, right? So those companies, they don't require technical knowledge based on type of product you build. Okay. Let's say if you work on like Instagram filters, they don't even care, you know how to code it. You just need to know what millennial wants or Gen Z wants, how to create different filters and how to engage with influencers, how does influencer operate, right? So that's the knowledge and your people skills are way more important in technical skills based on the type of product you build. Okay. So therefore I recommend you do not go to Google, but 90% of the jobs out there, you can be a product manager, even if you don't know how to code. Now, how deep the tech knowledge you need is that you just need a general understanding of different tech products. You can talk about it. You can understand why certain product is going to be successful, what technology they use, like technical one-on-one courses. I have lots of free courses. I recommend you guys to take and let me share you my screen. Right. Yes. Me crying. So funny. This video. Cool. La, la, la, la. Where is the thing? Okay. Let me go to my home. Okay. Hold on. One sec. Let me share with you the technical courses, how deep you can go. So right here. First of all, watch this video regarding do you need tech background to become a product manager? Go here. I break it down based on different type of companies. Hold on. Hold on. Hold on. Let me find how to become a manager with no experience. Same thing. Right here. No tech background. Tech coding, coding, coding. Yes. Is coding required? Watch this video. Okay. I break it down based on different kind of companies and how deep they go with and the technical knowledge within the interviews and where you need to look and like. So watch. Make sure to like this video. Like, comment, subscribe. Let me put this link in the chat. Okay. Right here. So watch this one now. Let me share you the three courses. So how deep you should go. Okay. That's not that deep. Easy. So come to this video. My most popular video. Top 20 Google interview questions. And come here. In the description, I have lots of recommended technical software architecture classes. These are like very easy eight tech courses for you without a coding background. Computer architecture tutorial. 15 minutes. Cloud architecture tutorial. How does the web work and all the like free YouTube training tutorial. You don't need something very deep. Okay. Very important. You just need to understand how do they work so you're able to communicate with the engineer team very efficiently. So come and go to this video. So this has all the free resources ever need to man you want next job and top attacking stuff. Copy the like this video. 600 likes. How many comments? I'm very curious. How many comments? 26 comments. So come watch this one. Now let me put this video in the chat. So you have it as well. Cool. Now let me let me let me go back to your question. Because see lots of company. And now another thing I want to recommend you guys is lots of companies, they mentioned one tech background, but you search the wrong job. Okay. So come here. There are 20 different kind of PM job you should apply for 20 jobs with 20 jobs. Right here. 10. Sorry. 10 jobs. 10 PM jobs you should apply for. Make sure come to this video. And I break it down regarding many different types of jobs out there, like all the introductions, gross PM, B2C PM, you have no technology background. It can be a gross product manager. Okay. Also be like B2C product manager, right? Many different kind of operations manager, APM social product managers, all of those are very relevant to you based on your background. Okay. So come to this video and study and make sure to like this video, like share and comment on this video. Okay. I had 27 likes already. So share this one. Let me put this in the chat so you can watch it right away. One sec. Oh, now we got CSD. No, you don't need CSD degree. I don't even have CSD degree. My PhD is in material science. I don't even know how to code. But I work on most technical product. I work on AI product five years ago. And lately I work on edge computing product in the cloud space. And I led the team of like very senior product managers and software developers to build very technical product and no problem. Okay. So you can do it as well. Just need to take those entry level courses I recommend in the video. Do it ASAP. Okay. Like one hour total, something like that. Just watch it. Take those entry level classes. You do need it. You do need entry level, but you don't need to be that deep. Okay. Cool. Let me, oh, I have a pre-submit question. Hang on. There's one question submitted here. One sec. Oh, there's one question. There's one question. Someone asked me this question. How to get job from beginner to advance. Okay. Someone asked this question. How to get job from beginner to advance. I assume this means beginner to advance, which means you're already a beginner product manager. How to become a senior product manager. Okay. Awesome. I love this. Actually our, our bread and butter is actually how we get promoted after you land a job. Now how to land a senior PM job. First of all, I do want to show you exciting news. We have student Jackie. She's also a mentor in our program right now. So Jackie, she became associate product manager in our program a year ago. Now, then then was in like half year she became sugar promoted as a product manager. And then within four months after being only being a product manager, she learned to senior PM position. How she made it happen. Do the following three steps. Number one, if you already have a product manager title, but your entry level is important to do a great job in your current job. Number two, you need to increase your scope of work scope impact as a product manager in your current job. Number three, you just need to know how to package your stories and also like conquer the like the concern from the interview. People say, Oh, you're only product manager for like four months. How can become a senior PM? Right. People will say that you're too young, but age is not a problem. You know, it's not about like too young. Who says so? Like lots of young people. Mark Zuckerberg started his Facebook when he was 21 or even during college, right? Quit Harvard, you know? So age is not a problem. But how would you respond? In A, I am young. I'm probably the youngest senior product manager, but this is the impact. This is how I can help you to grow your company. I deserve this much, right? In terms of getting paid more as well. So you following the three steps, you're able to become senior product manager as well. If you're interested in PM, et cetera, regarding we go in depth, regarding those three steps, go to drnancy.com and learn more. You can make it happen as well. Okay, submit application, see the qualify for the program. I believe this is all the questions I have. It's all the questions I have. Okay, someone said this question. Rika said, I've done basic medical and want to go ahead. I see medical training, want to go ahead in PM and get HMB sponsorship as soon as possible. Yes. Okay, so and actually lots of students in our program need HMB sponsorship. We give HMB strategies or OPT strategies about all the visa stuff. We know in and out. It took me 12 years to land a green card in the US forever. Okay, so I learned through my mistakes. It's crazy. So yes, you can make it happen as well. Yes, we talk more here and today we don't specialize in HMB. Maybe next time I have a separate YouTube live and talk about HMB. Okay, so let me know. Do we answer all the questions? Double check, double check. Let me see. Comment below if you want me to have a YouTube live talk about HMB. Comment below if you want me to talk about that. But let me double check. Do I have answer all the questions? Yes. I answered, I believe I answered all the questions. If not email my team. Okay, cool. The email is also in the deck right here. Let me send you email. Email my team. Email my team. Email the team at info at PMExcelerator.io and you are able to get also like information resources from us and then come here. Okay, and anybody who is interested in PMExcelerator coaching program, come to our free masterclass on our website. Help land and keep your dream PM job during hiring freeze. Go to our website, doc.nancy.com slash masterclass and stay until the end to learn more about our coaching program. And over there you're able to quickly know what to put on your resume, help land more job opportunities in our free masterclass as well. Go here and start learning today. Awesome. Okay, great. I believe I answered all the questions and thank you for joining me. Happy holidays, Merry Christmas and Happy New Year and talk to you soon. Make sure to share this video and share any videos, share, comment, like and this doc needs to be talk to you soon. See you guys.


Gandhi supposedly said be the change you want to see in the world and this is really a summary of a larger speech with the important line being if we could change ourselves the tendencies in the world would also change. So today we're going to look at a day at the Institute as a jumping-off point to discuss the change we want to see in the world. This is the Lotech podcast. Hello and welcome I'm Scott Johnson from the Lotech technology Institute your host for podcast number 57 on October 14th 2022 coming to you at the Lotech Institute's recording room in Cooksville, Wisconsin. Thanks for joining us today we're taking a look at what a typical day is here at the Institute but again every day is pretty different and don't forget to follow us on Twitter our handle is at low underscore techno like us on Facebook find us on Instagram subscribe to us on YouTube and to check out our website LotechInstitute.org there you can find both of our podcasts as well as information about joining and supporting the Institute and its research. Also some podcasters should put ads on podcasts unless you hear me doing the ad someone else is making money on that advertising while all of our podcasts videos and other information are freely given they do take resources to make and if you're in a position to help support our work and be part of this community please consider becoming a monthly supporter for as little as $3 a month through our Patreon page patreon.com slash Lotech Institute if you'd like to sponsor an episode directly please get in touch with us through our website LotechInstitute.org it's now 655 on a Thursday and I'm down here in our living room slash breakfast nook you can probably hear my one-year-old daughter in the background banging her plate as she enjoys a banana I've been up for 45 minutes getting myself together and one of the kids up we'll have to get the other one up here in a minute we live here at the Lotech Institute or better said the Institute lives right now at our house that's something we hope to change in the next two years we hope to make a permanent home for the Institute in a barn with a workshop in an office but that's not where we are yet and so a lot of what we do is in our own home and in our garage where we hold classes in our driveway and on our property if you ever run a business out of your home you know that it can be a challenge both interpersonally with other people who like to have privacy in your family but also just you know insurance liability wise and all that all those sorts of things so it's something we're working to change but we also enjoy the doing a lot of the things we talk about both on the podcast on our YouTube channel and just on our website as you'll see as we go throughout the day much of what we're doing at the Institute connects somehow to how one lives their life and since we're living here we often get to enjoy and suffer from the Institute's projects today on the menu is cinnamon raisin toast that I made chased work at a bakery so that's fun some of the ingredients in the bread our ours but we've already had to plant most of our wheat seeds for next year and we've given away and we still have some to grind up and use that's what we try we try and integrate as much as we can our own food and eat everything from our own property as we can and then on top of that we supplement with exterior our incoming foods on the grocery store and things like that for example or we don't have any dairy animals we have two growing kids so we certainly buy milk and cheese and dairy and things like that because we can't make them on a scale without access to a dairy animal which is kind of a shame because we live in Wisconsin and I do know there are neighbors that have dairies and in theory we could walk over and get some milk but farmers here don't want to give raw milk to people they don't necessarily know and so it's hard to bring that up with people and say hey can you do something illegal for me when we only know each other in passing so that's kind of a thing that might need to change in the future if people need to seriously think about changing their food web from something that is national or international really to something that is hyper local something you know could you walk and get all the food you need probably means less dairy for most of us but there's certainly a lot of food around you and so that's something we often think about if you haven't watched our Food Mageddon series on YouTube you should head over there and check it out that's where we spent a year where we only ate food that we grew gathered or hunted near us so that's a yeah it's interesting place to start if you're kind of just joining us or if you haven't seen that so breakfast is toast sometimes we have eggs from our chickens sometimes we'll have granola that we've made for the most part we try and at least integrate a lot of our own food into it that's water going into our tea kettle we use a reverse osmosis filter because we live in the country and the nitrate levels are really high because all the industrial ag land around us dumps tons and tons of nitrates onto the fields it's amazing how much nitrogen we've fixed using largely natural gas and other power to compress atmospheric nitrogen into usable nitrogen that just gets flushed into the environment because we over apply it but so yeah we have to use a reverse osmosis filter out here although if I lived in the city I'd do the same thing for other reasons Institute runs on tea not coffee someday I'd like to figure out how to grow my own tea leaves but for now importing dried self-stable goods is really not that bad you can bring tea and other dried commodities really long distances by ship which is much easier on the environment than other types of commodities especially ones that are fresh or need to be kept refrigerated cool or moved very fast so yeah if you think about what you're buying at the grocery store things that are shelf stable and do not require quick transportation from a long ways away are generally easier on your carbon footprint okay bye have a good day you too bye penny and the whirlwind is past kids are gone for the day three days a week they're in full time which allows me to do three days a week of work the other days are juggling and now a few minutes of computer work updating blog post answering emails answering YouTube comments and taking care of things that can't wait and double-checking that I don't have any appointments or things I'm gonna miss throughout the day just a bit of calm after the storm of getting the kids together and out the door now I am chopping up peppers these are hatch chilies actually and I am popping them in the dehydrator because at this time of year the dehydrator needs to basically be running constantly to dehydrate peppers and apples and tomatoes and everything else okra yeah whatever we can get in that dehydrator we get in there and get it going because winter's coming as they say before I went outside to get some work done I had to pop down here to the basement and I am shoveling finished worm compost into a sifter because I need to make some potting soil because we need to get our winter herbs going we have an herb spot in our kitchen and they grow in pots so my mom is gonna get those herbs started today and also we need to start some lettuce because our winter cold weather greenhouse will be starting up here soon so we gotta start some cold weather leafy greens like spinach and lettuce and other things so that we can have fresh greens into the winter now to open up the chickens give them some scratch fill up their feed or a voracious little dinosaur descendants here unfortunately we have a sick one who has something called blackhead which is primarily a turkey disease but because the medication you would give to it would make it unsuitable for human consumption the medicine has been banned and so my sick chicken is gonna die because I can't get the medication legally in the United States so there you go kind of rare in chickens and this is my second chicken in four years that's died of it that's unfortunate we've had Hock attacks this summer but overall we have a decent flock here we're going to do our fall butchering in a few weeks and bring our flock number down to about 10 for the winter all right so I've just planted out 50 spinach cells and I'm watering them in which is the spring water you hear in the background completely soaked then they'll go in our oven where the pilot light will warm them up and sprout them and they get moved to our greenhouse where they will grow over the next few months first under sunlight and then under grow lights using crapwood to heat the fireplace in there it's a constant cycle of starting seeds harvesting and starting seeds and harvesting on a rolling basis I now find myself 20 feet above the ground on top of a greenhouse repairing a cut in my neighbor's greenhouse that I also get to grow in so I'm not gonna record much up here because it's dangerous but fun just another random thing I get to do today now I'm inside the greenhouse picking tomatoes these are Roma style tomatoes and Marzano twos I'm picking these these are a paste type tomato these will be turned into things like salsa and marinara sauce and tomato paste and sun dried tomatoes and on and on and on I also need to get some seeds from lettuce that bolted in here beginning of the season so I can plant that out to our other greenhouse which is more cold weather oriented now I'm in the garden taking up potatoes never ending potato harvest continues do a bit when I have free time usually we get around 200 pounds of potatoes a year which is enough to get us through the next season if you want to hear more what I have to say about potatoes you can check out a couple of different podcasts in our feed just scroll back you can find like potatoes will save humanity and how we plant potatoes based on the USDA study we ran so anyway yeah try and spend a lot of time potato field in the fall and now it's time for lunch at almost one o'clock supposed to be noon toasted tomato sandwiches with tomatoes from the garden and bread we made earlier this week and working on the computer trying to get stuff done sort of a quick break and then back to it and now after lunch I am spending a quick half hour in the office moving papers around and answering emails again and just doing general office work which I honestly don't do enough of and if I had more time I would spend a little more time in the office but that's how it goes especially during the summer spring and fall when there's a lot to do out in the gardens and fields now I'm out in the wood shop moving around staves that are going to be made into buckets this weekend at an event that I'm demonstrating coopering at so I'll be making a little buckets and pails my kit there at the New Glaris Heritage Festival or Harvest Festival at the historic Swiss Museum in New Glaris Wisconsin so I'm looking forward to that it's always a fun event every year so I'm now prepping all my wood for that I have to get some pine boards sawn down to approximate shape and then I'm also going to split up a whole bunch of ash to make buckets completely by hand so always something onward and just like that my free time is done it's now 10 to 4 so I have to go get the kids from daycare in school etc get them home get them fed get them doing stuff oh and here comes my neighbor with basket full of produce for me okay all right now I have just a few minutes while the kids have a snack to scrub the back porch in order to then put some linseed oil on it just to get the muck up because this is a muddy place because we come in out of the garden and it just builds up over the year so it's good to wash it off get a good coat of linseed oil on it before the winter comes now we are with the help of the munchkins I'm watering the greenhouse getting the beds ready we are watering in the greenhouse I don't know why we're all coughing we're watering the beds in our winter greenhouse to get them all nice and uh-huh water level is all nice and built up so that they can accept the spinach and other things that we planted out today so just kind of getting this winter greenhouse in shape you can see this winter greenhouse called a wallopini on our YouTube page you can see the first half of the build so check that out if you haven't found our YouTube page yet last thing of the day is covering my tomato plants and other sensitive plants with row cover because we're supposed to get a freeze tonight so I am just covering up with my my daughter my one-year-old here on my back in a backpack and we are out here just pulling the row covers right over so that they stay just above freezing and my son is at the other tomato patch eating any ripe tomatoes so they don't go to waste of course before I go and cover that one last thing of the day and then we'll be making dinner and having family time as you can see each day here is filled with bits of work sometimes we spend all day on a project and others we are moving from task to task like today but every day is different and much of what we do is building up infrastructure or growing food in a future with less abundant energy this is a model that most people could and frankly would have to adopt we wouldn't be able to rely on long distance transportation networks as we do now for fulfilling our every desire by mixing physical and mental work it gives me for example a more balanced feeling than if I were just behind a desk all day or working outside alone thanks for following along today and I hope this gets you thinking about what your life could be like if you began to disengage at least a little bit from the large-scale economy that really can't be kept up indefinitely that's it for this week the low-tech podcast is put out by the Low Technology Institute the show is hosted and co-produced by me Scott Johnson and co-produced and edited by Ines Hizuki this episode was recorded in the Low Technology Institute's recording room subscribe to the podcast on iTunes Spotify Google Play YouTube and elsewhere we hope you enjoyed this free podcast and if you'd like to join the community and help support the work we do please consider going to patreon.com slash lowtech Institute and signing up thanks to our forester and land steward level members Marilyn Scarpon, Sam Braun and the Hamuses for their support the Low Technology Institute is a 501c3 research organization supported by members grants and underwriting you can find out more information about the Low Technology Institute membership and underwriting at lowtech Institute.org you can find us on social media and reach me directly on Scott at lowtech Institute.org our intro music was cobwebs off the album headless horseman by Halizna that song is in the public domain this podcast is under the creative comments attribution and share a license meaning you're free to choose and share it as long as you give us credit thanks so much take care and happy Halloween if I don't get to you before then


All righty. Welcome to my talk. My name is Tim Allison. I've been asked to speak at the podium, so I will stay at the podium. I will not wander around and be a dynamic speaker, like I've been trained to be as if I've had training. Anyways, I work at the MITRE Corporation. I'm a committer on Apache Poi, PDFBox, and TECA. So off we go with evaluating text extraction with Apache TECA's new TECA eval module. This is where the next slide happens naturally. Yay. Okay.  My debt of gratitude is rather large. First to David Smiley, committer on Apache Solar, who first got me off of my laptop and into open source. Without him, I would definitely not be here. Nick Burch, of course, brought me in even though my first patches were done in Notepad and Groovy converted to Java. So sorry and yet thank you so much, Nick. Same for Chris Mattman who fostered my early stuff and has been a fantastic collaborator on TECA. Now Tillman Housher on Apache PDFBox has been a great colleague in working with, especially this eval stuff and helping us figure out what metrics we want to use in the eval. Dominic Stadler from the Apache Poi project has been really helpful in bird of a feather in gathering common crawl data and running large scale regression testing. And of course, all of my other fellow devs and users on Apache Commons, Poi, PDFBox, TECA, and the entire ASF community, thank you all. This is a marvelous community to be a part of. So those are the people. Also hugely indebted to Common Crawl, the Common Crawl project from which we've gathered a number of documents so that we can run our regression testing and also the GovDocs1 corpus which was gathered a number of years ago. And then RackSpace has kindly hosted a VM for us and I'll talk about that public VM shortly. So anyways, I'd like to start by saying thank you to so many people. So an overview of the talk today. I'll talk a little bit about content and metadata extraction in case anybody somehow doesn't know anything about TECA. This is the TECA Medium or TECA 200 class, not the TECA 101, but I'll go over a little bit of what it does. I'll talk a little bit about the motivation for TECA eval. I mean, what could possibly go wrong with text extraction? I will tell you at least some of the things I've encountered. And I'll talk about an overview of what's in the new package, the workflow and using it and then I'll go back and forth a little bit and talk, share a little bit about our terabyte public corpus that we use for regression testing before the next version of TECA, before the next version of POI and also PDF box. I'll also talk about limitations. I do not have an easy button. I've seen the word magic in two talks earlier today. I have no magic, sorry. I also, yeah, so if you came for magic, I don't want to disappoint. It's still early. David North is giving a fantastic talk on Apache POI in the room over there. So if you're looking for magic, that might be a better option. All right, so you might ask if you are paying attention to these things and looking at my slides from two years ago, well, what's really different from your talk two years ago? Well, now it works and now it's actually integrated into TECA and it will come out with the next release, which we should be kicking off tonight or tomorrow. I think we're cleared to go now on TECA 115. So a lot of work has been done, especially iterating with Tillman-Hauscher on PDF box to improve evaluation. A lot of things have been improved and it now basically works. So first off, content extraction and human language technology. For folks who do more fun things like entity extraction and search and machine translation and whatnot, the stuff that we do on Apache TECA is really boring. We get stuff from all sorts of different file formats at the bottom over that red line so that they then have text so that then they can do the more interesting things like search or entity extraction and so on. But it's a challenge to get all of those different file formats, figure out which type of file you're looking at and then apply the appropriate parser. A nice thing about Apache TECA and the goal of it is to have the same interface no matter what type of file you aim at it so that we don't all have to go and reinvent our own file ID and then figure out which parsers to apply. So the whole goal of Apache TECA is to get from bytes at the bottom to something that we can then process at a higher level. Speaking of which, I learned that this week is National Infrastructure Week in the US celebrating infrastructure projects that nobody pays attention to or nobody cares about. So this is not deep learning IoT stuff but it is critical to those kinds of things because if content extraction fails, those other downstream more interesting and more exciting projects will also not fare well. So typically high level components of a media processing stack, you start out with files or structured data then you throw those into your something and of course this is multi-step of collapse this because I'm interested in the bottom part and then you have a user interface on top of that and I'll come back to this shortly. But let's not forget about metadata. Another useful thing that TECA does is it will pull out metadata that's embedded or stored within the documents. So you can often get the who, at least who said they were, who self identified as the author. You can get digital signatures, you can get company from emails, the from twos and so on. You can often get hardware versions or names, software versions or names. You can sometimes get globally unique file IDs if your file happens to have XMP in it which can be quite useful for some use cases. You can also get title and keywords. Sometimes you can even get geolocation. So for images that have latitude and longitude you can extract that. Sometimes even you can get the original file location for where it was last saved on somebody's hard drive. Or sometimes you can get that for embedded images in a file. I thought that Microsoft had been moving away from that but I just discovered recently that Microsoft has started putting that back into XLSX files in their future-ish thing for OOXML. So we can now extract where somebody last saved their XLSX file which for some applications may be useful. Then also of course when when it was created last modified and printed. So beyond the standard types there's all sorts of fun custom metadata that you can pull out of files if that's the kind of thing you're interested in. If you're into that kind of thing. So this is what I call blood on the highway which is a reference to driving films from when I was younger or actually from before my time where people were being scared straight about not drinking and driving or being careful while driving. When I talk about these things these kinds of things don't happen all the time in Tika. Tika actually works really really well but given that I sit and watch the Jira a whole lot and given that I run Tika against a whole lot of files I see a lot of things that can go wrong with Tika so that's what you're hearing from me today. What you should not take away from this is that Tika is a total disaster and you should run from it. It actually does work quite well but this is more of the kind of attacks on me when things go wrong with text extraction coming up. So in this example application I focus on search which is one of the more basic and primary things that people often do with content that's pulled out of files. What I've seen in a number of places where I've worked is we have kind of content metadata extraction or data from a structured data store search system and then we have that user interface and the amount of attention that's applied to each layer at least in the couple of places where I've seen these kinds of systems is all at the user interface. As long as somebody can enter in search term and some documents come back, the users are happy, the managers are happy, the GUI developers are happy, it works, everything just works, this is marvelous but where I don't see as much effort put in is any regression testing or any testing on the lower parts of the stack which frankly aren't that interesting and we just, it's oftentimes that people hope that the components at those levels are just working and they trust that those are working often without checking. So as you know, when things go wrong with the foundation, stuff happens. So what can go wrong? And here's my taxonomy of things that can go wrong, not to scare you away, but one, we have completely expected exceptions. Expected exceptions, yes they are expected exceptions. So if you have truncated files, our parsers are not designed to handle truncated files. Some handle some types of truncated files better than others. If it's a password protected file and you don't have a password, we're not handling it. If the format version isn't handled or if the file type isn't handled, we aren't gonna be able to do anything with that. And sometimes they're just plain corrupt files for whatever reason and we're not going to be able to do those. And of course there's a spectrum of corruption. So it could be the case that the main application that's associated with it, Adobe Reader for example, can get text out of a file but we can't. The flip side also happens sometimes the PDF box is able to get content out of a file where Adobe Reader is not able to. And sometimes you try to open it up and you just get nothing.  And then you have the somewhat expected exceptions. So a parser has a problem with a non-corrupt file. The code base for Tico, we're now at 50 megs-ish of a jar. There's a lot of code in there and a lot of moving parts. We don't have eyes on all of the code all of the time. Each dependency is constantly being upgraded and improved. So we don't have a chance to look into all of that code and follow all of that to do rigorous code reviews of what's coming into Tico. So it happens that parsers sometimes have some problems and hopefully they will throw an exception and you'll be done with it and everything's fine. And you can log that exception and hopefully open a ticket on the JIRA, get it fixed and then it will be fixed in the next version of Tico ideally. And sometimes there are just corrupt files that nothing can handle and there's not a lot that we can do about that. All right, so note that when you do get exceptions, especially of the somewhat expected exception category, you may get some text, you may get some metadata, it all depends on the parser. So if you do get an exception, don't necessarily throw out whatever you got from the handler, that may still be of use for some use cases, search, for example, or other things. So those are the basic problems. We also have catastrophic problems. Now these happen really rarely, fortunately. And the good news is that when they happen, you often know about it. Not always, but sometimes you do. So out of memory errors. And out of memory errors can be not so bad sometimes. Yeah, we'll corrupt your JVM and it could be a real problem. But what really gets interesting is when you're approaching in OOM and Java's, the garbage collector is just trying.  But from my perspective, if I get one of these files off the JIRA, it's an entertaining thing and exciting, don't get me wrong. The other is sometimes you can get it out of memory error from a four byte file just because something went wrong with the parser. There was a misunderstanding in the parser and you can get it out of memory error. The other, when you get those at least, you know something's gone wrong. Slowly building memory leaks are a little more interesting unless you're doing profiling and really paying attention. You often don't realize what's going on. So again, you can get garbage collection issues where you're just running five threads. How could you possibly be pegging your quad core machine? It may be a slow building memory leak and the garbage collector's working on all threads or multi-threaded to try to clean up some of that memory. Permanent hangs are a joy to behold. That's also, that typically only results in a single thread hanging. It won't often corrupt or take up the entire machine. But when those hit, those can be quite exciting. Antica 1132 has some nice descriptive narrative for you if you care to follow about what happens when a parser just goes into an effectively an infinite loop. That can also be a real problem. We also have had a couple of security vulnerabilities which we fixed in one, at least by the ones that we're aware of, we fixed by 114. So we had a couple of XXEs which we fixed. We also had this great arbitrary code execution vulnerability so that somebody who carefully crafted a MATLAB file would be able to run whatever software they felt like on your computer that was running Tica. Which is really exciting in the wrong way. So again, this is blood on the highway. These are extremely rare issues, but these can happen. And if you are in production, if you are handling, as Nick says, millions of files or billions of files from the internet that you don't trust necessarily, these things will happen. And if you are running Tica in the same JVM that you're running your indexer, these things can cause real problems. So please do not do that. Or just beware that these things can happen. All right, so those are the kind of expected exceptions, the really, really rare exceptions, and then there are the hidden problems that you're not really aware happened. And these can be, you can get garbled text out of the content extraction from slightly garbled to totally hosed, and I have some examples of that. We also have missing texts where you have just blocks of text are missing from the document. You would expect to get much more text if you open it up in the native application that's supposed to handle that file.  You can miss, you can be missing attachments thanks to a bug that I added and then fixed just in time before 115, before the process for voting for 115. And also, if you are using the classic Tica handlers, which just pulls text out of documents or XHTML, the default there was to swallow embedded file exceptions. So if you had a zip file that contained a whole bunch of Excel files, and those Excel files all happened to be generated by the same source, and that particular version of Excel was a little bit different from most, you could wind up getting exceptions on all those files and you would never know it, because in the default use of Tica, we do not let you know that there was an exception on an embedded file. One batch of documents I recently reviewed show that they had 50% of their Excel files were getting exceptions, and the folks who were running Tica had no idea because they were running the classic method which has no warnings about these things. So those are things where you're not getting an exception, but something could still kind of go not as well as one might like. So examples, examples, here we have the highway. So this is what happened on one file when we upgraded from PDF box 1.8 to 1.86 to 1.87. We were getting the text on the top and then when we upgraded, we got the text on the bottom. It's a complete single substitution cipher so that capital T goes to capital B, lowercase a goes to capital G, fine. This would pass Zipfian distribution beautifully. It would pass word length statistics beautifully because all the words are the same length, but now you're throwing all of these brilliant words into your solar index, your Lucene index, and now you're blowing out your, of course solar's strong, Lucene's strong.  So it's two great things that happen when this kind of thing happens is you're making your index suboptimal and also people can't find the documents they need. This is an example of missing text and this was one of the issues I think that initially got me out of my notepad developing laptop and into open source and Nick put up with so much on this little patch I had for this particular issue. This is this poor person's CV with lots of great detail about what she does and that's all the tika pulled out. So this is mildly amusing because it doesn't look to us like it has real world consequences, but if you think about recruiters or anybody in that company looking for particular keywords, they're not gonna be able to find that for this person. So these can have real life consequences when text extraction does not work. And again, you can run all sorts of metrics on this and it would pass a number of them because it's basically, you're getting English-ish stuff out of there. You're getting some text out of there, but you're not getting the full document. So this is an example of missing text. Some other things that can happen is you may get some noise. If a PDF has been scanned and it has embedded OCR in there or text that's been generated by OCR, you could get some noise in there and we all know and expect that kind of thing sometimes in PDFs. But this was a great example where you have the image, you have the text that was extracted and you can prove to people that you can find this exact document if you search for II-onitoring on Google because Google at this point was relying on the text that was stored in the document. In this case, it happened to be OCR. All right, so basically, my main point on this is that if you take whatever you get out of teak and throw it into solar and then throw a great user interface on top of that and hope for the best, you don't know what you can't find. So please take a little bit of time if possible to do some kind of evaluation on the content that you're getting out. So that's the blood on the highway chunk of the talk. For some examples of things that can go wrong, again, Tika works quite well a lot of the time. So please don't go and build another Apache project that pulled content out of documents. All right, so the dream from all of this is Tika 1302. So my dream was I was motivated by all of this stuff of the above. We have only about 1,000 test files between Poi, PDFBox, and Tika. It sounds like a fair number, but it really isn't given the amount of marvelous things that can happen in PDF files and also in Microsoft Office files. And I'm just talking here about PDF in Microsoft Office. Of course, we handle a number of other file formats. So 1,000 test files sounds like a lot, it's not. The other motivation behind Tika 1302 is that some groups made the mistake of giving me right access to their projects. So unit tests are nice, but against me, come on, you're not gonna cover all of the stuff I'm gonna ruin in your code base. So this was another motivation for why I had the dream of Tika 1302. So my dream was to run Tika against a much larger corpus, either nightly or weekly or something, and then automatically recognize regressions. And this seemed like a great thing. We have regression testing, we have continuous integration, we have all these systems set up, that's great. There is no magic, and I'll talk about that shortly. So that was the dream. As part of this dream, one of the components was some evaluation metric that could say how well did text come out of here, or how well are we getting text versus how well were we getting text with an earlier version of Tika or a different tool. So Tika eval, let's focus on that now for now. That's the component within the larger dream of, okay, so you've extracted some text, how language-y is it? Do you have some kind of sense of how well you did? Or can you compare two different groups? So this, as I mentioned, is available in, or will be available in Tika 115, which should be coming out shortly. Okay, so a high-level overview of what's in the Tika eval module. It does not run in Spark, I'm sorry. So it's, I don't have any of the cool hip things that all of the kids are using nowadays, but it runs from file share to file share. There is no bat scripting involved yet. There's no pearl here, but it does run file share to file share. And the notion is if you do have a modern document processing pipeline, Spark and so on, then at least with Tika eval, you can do a random sample of what you have and run it file share to file share. Or, as I mentioned, our JIRA is open, the committers are standing by, so if you do want to add an integration point for Tika, or even just share lessons learned, let us know. As Nick pointed out in his talk, one of the great things about Hadoop is it will try files again and again, and unless you tell it to stop trying files, you could run into some problems. So you have to be careful in the large-scale processing frameworks. So the scope of Tika eval is really quite humble. It's file share to file share. And there are basically two modes. One is profile, a single extraction run. So if you run Tika against a batch of documents, and you have a parallel directory structure with the original binary documents and the text that was extracted, you can run a profile on that and say, how many exceptions did I have per mime type? What languages were detected? What's the number of words per page ratio, which can give you insight into whether your PDFs are image-only perhaps, or it can give you other insight? So there are a number of things, and I'll talk about that. The other big mode is two extraction runs. And the notion here is, hey, if you have ground truth, great. If you're running an OCR study and you have ground truth for what text should have been OCRed, you can use Tika eval the compare mode to run ground truth against what you're currently getting out. Or you can experiment with different settings in your OCR to see which one overall gives you better performance on your batch of documents. You could also be comparing different tools. And I'm not saying that there are other tools besides the one Apache Tika to pull content out. In the Python world, for example, there wouldn't be anything similar to that. But you could, if you wanted to, I guess, compare other tools to some of the things in Apache Tika. Or you could compare tool A with settings X, tool A with settings Y, or even just two different versions of Apache Tika. So when we say that we have a new, great new version out, you can run Tika eval on your documents to see if there are major regressions that would prevent you from upgrading. And when that happens, please do open tickets, and I'll talk a little bit about that. So that's a high-level overview of the scope and basically the two modes. Before I go much further, I do have a couple of definitions that I've come up with on, or excuse me, that the Tika eval community has come up with, and the community would be a bus quotient of one largely. So the original documents or container documents, those are just the binary files that you aim Tika at. Whether it's an actual text file or whether it's a word file that might have attachments, I just call them original documents or container documents. Embedded documents are anything that show up inside that document, which are basically viewed by some people or many people as an embedded document. That could be, for example, an attachment on an email. It could be an actually embedded file. You can embed a doc file inside a PDF and then put a zip inside that doc file and do all sorts of choice code, doll things. Or it could be a file format that doesn't really ever exist outside of being an embedded document like EMF or WMF or XMP or XFA. So some file formats really don't exist on their own. So by embedded document, I mean anything that shows up inside a document and is basically recognized as an embedded document. Another term is extract. So an extract covers anything that you've pulled out of a document. There are two basic modes in Tika eval. One is text. So this is useful for if you're running another content extraction tool and it just pulls out text. You can have the text extract, what text was extracted from that. We also have this recursive parser JSON stuff and I'll talk about that in the next two slides. Tika eval was basically designed for the JSON format but based on Apache Con from two years ago, I added the dot text handling option. Yeah, so the details on handling dot text files or as extracts are all on our Wiki. All right, so why the recursive parser wrapper? So let's say that some person has taken an embedded four dot text file, thrown that in a zip file, put that in another zip file, put that in another zip file, another zip file and then put that in a Word file. So now you have this massively hierarchical bunch of documents all in a nice Microsoft Word file. In the classic Tika extraction that is handled as one collection of sacks events and the way that embedded documents are handled is you have a div class for that embedded file, embed1, embed1a and you get the embedded path for where that embedded file is and then you get the text that's extracted from that. And given the framework of streaming, that's kind of what could happen. That's what you can do with embedded documents. But one problem with this is that the metadata from those embedded documents is lost. So if you have a zip file of image files and those image files actually have latitudes and longitudes with the old version or the classic version of using XHTML, all of those lat longs could no longer be processed and you'd lose them. The other thing, as I mentioned, we swallow embedded exceptions without warning you at this point. So that's another area of concern. Yeah, oh, the other thing is that sometimes with classic XHTML, we try the best we can to get the metadata out of a document before writing the XHTML but sometimes we can't get metadata until we get further down the parse of the original document. So it could be the case that you get metadata extracted from a file but it doesn't show up in the XHTML. So to solve these problems, I stole code from Nick and Yooka and put that into Tika and we're calling it now the recursive parser wrapper. And you can get to this through Tika app. It's the capital J option or R meta at the endpoint in Tika eval. And this gives you a list of metadata objects for each embedded document and then each metadata object has the content type or it has all of the metadata with a special metadata key of XTV content and that's the actual text that was extracted from the document. So this will maintain stack traces for embedded documents and it will maintain content for all of the embedded documents like we had in the traditional but it will also maintain all of your embedded metadata so you can get all of your latitudes and longitudes if those images are put in a zip file. So in Tika eval, everything is built around this but we do have a way to handle regular dot flat text files for extracts. So the workflow for profile, you generate your extracts. In my case, it's used Tika in batch mode so Java, JAR, Tika app dash, I for input directory, O for output directory and then we have a parallel directory of input files and extract files. Then we run the profiler to populate an in-process H2 database and there's the Java command for that. You specify the extracts directory and what you wanna name your H2 DB. And after you do that, that calculates a number of statistics, puts them in the database and then you can dump the reports and the reports are driven by a whole bunch of SQL that's stored in an XML file that you can modify. So you can choose which reports you want to run on that batch of documents. Tika comes with reports that I've found to be useful or SQL that I've run against the H2 that I've found to be useful and then those reports are XLSX files and then you can go rummage through those.   It's abysmal, but it's better than what we had. So I'm sorry, but if anybody knows JavaScript at all and wants to pitch in on Tika 1334, it would be really nice to have a user, yeah, it would be nice, somebody ought to be doing it. It would be nice to have a user interface for that because navigating through the directory, navigating through the reports can be a bit of a challenge at this point. Let nobody mistake, I do not believe that a bunch of Excel files is sufficient as reporting on this, but it's what we have. And next slide. All right, the other one, so that's profile. That's when you have a single run. The other one is compare. And for that, you run Tika on two different versions of Tika. You have an Xtrax A directory and an Xtrax B directory and then you just run the compare tool and that will compare Xtrax A with Xtrax B, pump all of that comparison information into the H2 database. And then you dump the reports from that. And again, you have reports. You have all of the individual reports you would get from the profiling mode for each of those A and B. And then you also have some comparison statistics that compare A with B. And let me talk about some of the features that we can extract or that we can get out of A and B. All right, I'm sorry, there's one other tool and that's start DB. So this just starts the H2 database so you can navigate to local host and actually interact with the database if you want. And that can be really useful, especially as you're developing the SQL for the reports that you want to develop. All right. So for the reports in the profile mode, we get count of metadata values, we get counts of attachments, we get mine counts for containers and embedded documents so you get a sense of what type of file types you have, what kind of embedded file types you have in your corpus. We get a fairly lengthy breakdown of exceptions. So counts by type of exception, whether it was a password exception or whether it was a runtime exception. Exceptions by mine type, so you can see PDFs were getting very small number of exceptions or jar files were getting a high rate of exceptions. We also have counts by normalized stack traces. Sometimes the message in a stack trace, we remove the messages of the stack trace so those can be collapsed so you can look for common patterns in your stack traces. And then we also have one report that shows you all of your stack traces so you can at least link back to the original file if you wanna do some digging. We also have in profile mode some utility, some stuff that helps you understand a little bit about the content that was pulled out of those documents. So we have language ID, we have token counts, we have this common words count thing which I'll talk about shortly. We have some statistics on word lengths which can be useful for some things. And we also have page counts. So if the file type has a notion of page in it, we record that. So for exactly the use case of which PDFs don't have that many words per page should we be looking to OCR those. In compare mode, we export all of the metadata that we did for profile but we do it for both A and B but then we also have some comparison. So compare the MIME counts for containers versus embedded documents. So we had a lot more DocX in A than we did in B did something change in our file type ID system. And we also have of course counts of MIME changes. And typically that a number of these things require human intervention or at least humans to interpret what's going on. And I'll get to that. We also have comparisons of counts by MIME type and a number of other things. For and then content and I'll talk a little bit about the content comparison. So before I talk a little bit about the content comparison let me step back a little bit and talk about this common words metric. So this was first proposed at least in our little group by Tillman-Hauscher. And the notion is just take a corpus count the most common words. For now we started with English only and we dropped words that had fewer than four letters I think. And just count the number of common words that you're pulling out of a document. And that divided by the number of alphabetic words gives you some insight. So if you're only getting 0.01% of your terms in your document are actually in the common words something may be going on. And I'll talk about how that might not be a problem but overall over a large corpus that should have roughly languagey kinds of things in it that can be a useful metric. We also did some custom removal of HTML markup terms like body and table and some other things so that if an HTML file was misunderstood as a text file all of a sudden we would get this huge boost in common words but it's all markup which is stuff we don't actually want.  So when you run with the common words and with the number of pages you can do some useful statistics on number of words per page. And you can also as I said do the number of common words divided by the number of alphabetic words to get some indication of how well you're doing per file or at least get some metric of files that don't look like the others. For content comparisons we have built-in similarity metrics. This is basically overlap. So that's of all of the words that were pulled out of document A, how many of them show up in document B divided by the total number of unique words in both of those documents. Or you can measure how similar they are including the number of counts. So if the word thus shows up 200 times in document A but only shows up once in document B the first metric would only look at the binary does the word thus show up the second metric takes into account that thus shows up 200 times in document A but only once in document B. So some other content comparisons we can look at the improvement in the common words score. And then you can do this per mime and say so for PDF documents are we doing better on the common words score now than we were before which can be quite useful to at least get a sense of what's going on. So this is an example of looking at 114 as we were moving into as I was doing some work to see if we had made improvements in 115 or at least major regressions. So here we have a single file. We had 786 unique tokens in TECA 114 1600 total tokens. The language ID was Chinese. Number of common words, zero. Interesting, okay. Then we have the top end tokens from that. I don't know Chinese so that frankly doesn't mean no good but I can at least plop them and Google translate and see if I get anything useful. The common words metric for that was 0% because we had zero common words over 1600 and three alphabetic tokens. Whereas in TECA 115 we're now getting 272 tokens so the numbers gone down. Who knows if that's good or bad. Language ID has changed. Okay, something's going on with this file. The common words now goes up to 116 for a ratio of 46% of the words that were extracted are now part of that common words list.  So this is an improvement and the goal is not necessarily to manually review all of your files and say that looks like a language I don't know or not but really to rely on that common words then to get a general view of how well you're doing when you're comparing things. This is an example of a small regression that we found in moving from 114 to 115 snapshot. This is, you can see that we had slightly more total tokens. We had slightly fewer common words. The overlap between these two strings is quite high. It's 95.5% of the words in A were in B. The top 10 unique tokens which is a measure of which words only show up in A and never show up in B. We get some good English-looking words whereas if we look at the words that only show up in B and never show up in A we get some things that show that something's not quite going on and it's probably the single quotes are being converted to I's so something's not quite working with the car set recognition in that file. As a human we can look at that and reverse engineer what the issue probably is. But you can see there is a small decrease in that ratio of common words for alphabetic tokens and there's a, sorry this should be, yeah the increase in common words is negative 89 or the decrease in common words is 89. So you can see that this was able to point me to a specific regression or a file that had a regression when we moved from 114 to 115. So we did actually, we did take this evaluation metric public. We now have thanks to Rackspace VM that is thanks to HDPD. We're posting files and results on that server. We have a terabyte of data in there. It's roughly three million files from Common Crawl and GovDocs One of all sorts of different file formats. We've highly oversampled for non-html, non-texty things. We're collaborating with PDFBox and Poi to run evals as part of the release projects for each of those projects and then also for Tika. This is really useful for new parsers and it's also really useful for that, hey I'm getting this parse exception but I can't share the document with you problem that we get all the time because now if somebody can give us a stack trace we can go look in our database of stack traces and say, ah, here's an example of what's triggering your stack trace and then we can go work on it. One of the Common Crawl tools we used initially was Dominic Stadler who works with us on Apache Poi and that's really useful for picking out specific documents from the Common Crawl corpus. So limits, yeah limits. So if you get more exceptions we have a problem. Well no, not always because sometimes you have a new parser when you weren't getting exceptions before because you weren't parsing those files or before maybe the parser was yielding junk and now you're getting an exception so that's actually good. Wait, we have fewer exceptions, that's great. No, you might not have been detecting that or you might now be failing to detect that file properly. So you're skipping it or now we're getting junk. More common words, great. No, actually, and this was, again, my fault, a serious bug that's duplicating worksheets. So we're getting more common words, great. Well no, you're duplicating your worksheets so that's a problem. Or you might get non-HTML-ish markup, it's still markup but not the stuff that we've removed from our common word list and that might be creeping through. Fewer common words, there's a problem. No, it might actually be a good thing so for all of these things that one initially thinking about would say, oh that's a sign of improvement or that's a sign of a regression. No, it can be, it often is but not all the time and all of these things require human interpretation of the data. So for Pink Floyd fans in the audience, the ticket has grown, the dream has gone. We can run these regression without ground truth, we will get some insight, we will know when things have changed, we will be able to drill down and figure out what's changed and try to make sense of that which is far better than what we were doing. But given that a human's needed, we really do need a user interface so please chip into that if you can. There's also the notion of collaborative tagging, we've been working on the same corpus for about two years now. Each time we look at the file, we can say hey, that's really good or that's a bad version or this file is totally hosed and we can expect to get nothing out of it and we should be doing collaborative tagging, we don't have a user interface to help with that yet. So the dream of Tika 1302 happened to hit reality but we're far better than where we were. So to conclude, text extraction is critical to many of our projects. Please evaluate, at least on a random sample of some of your documents. Please do not just throw stuff into solar and hope for the best. Please use Tika Eval if it suits your needs, please join our community and help us with the evaluation, with Tika, with content extraction, it's really quite critical. I have some resources and it gave a great talk on what's new with Apache Tika, also a great overview of what's in Tika. We have the Tika Eval Wiki with a bunch of other pages that points to Ryan Bowman, a fellow class assist who's worked on automatic evaluation of OCR or OCR where you do not have ground truth, has a really good post on what he's done, he has some really good ideas. So those are some resources and onward. Thank you so much. Any questions with that? Thank you so much for coming to this. Oh, please. I was just saying my mind's turning. I have a lot of work, great work. Thank you. Yes, please. Yes, fortunately, we have solar people in the room. I don't know if you've worked a lot with the data import handler. So there's a data import handler and there's a way that you can map from Tika fields to solar fields and there's a way to configure that. And I recommend doing that yourself because Tika can come up with all sorts of crazy fields and you do not want to use the schema less version of solar or the schema less setup in solar when you're importing from Tika. But if you look at data import handler or any other resources you'd recommend, Christine or anybody else familiar with solar? Okay. Yeah, yeah. Not that I'm aware of. I will say with the data import handler, that's to get people off the ground quickly so that they feel like they can ingest these things easily. But if you are handling a lot of documents that you don't trust, it's a really dangerous thing to do. So there's a fantastic post on using solar J to separate your JVMs so that you have a whole different cluster of solar versus Tika. But as Nick really hit home in his talk, Tika does a good job of trying to normalize those different metadata tags so that if PDF happens to call the person who created the thing, the creator, but Microsoft calls it the author, we normalize those to Dublin Core. Forget what is Dublin Core? Is it creator or author? Whichever it is. Whatever the Dublin Core is, we try to normalize both of those to Dublin Core. So we try to help normalize to the degree we can. There are some file format specific metadata things that we can't normalize. But you can, once you do the extracts, you can at least see what keys you have and then figure out which ones you want to extract. But it is also largely application dependent. Yeah, right. So you can run, yeah, java-tika-app.jar and then just put the name of the file. You'll get stuff dumped and standard out. Also we have a user interface. You can drop a file in there and you'll see what information you're getting out. I would recommend if you have a bunch of documents though running tika, perhaps tika-app, tika-batch, java-tika-app, iinputdirectory-ooutputdirectory, and then perhaps the j-option, so you get all of that metadata and then running some parsing on that to figure out exactly what you're getting across your corpus. Because just looking at one file at a time will not give you a sense of the diversity of tags available to you. So you do have to do some mining of your documents to figure out what you have with a huge consideration of what your information needs are at the search layer. Well, I'll be happy to stick around if there are any other questions. Thank you all so much. I know the feather left the building, so I'm thrilled that anybody showed up. And thank you so much.


probably shouldn't think of this as as like a win-lose binary outcome. Yeah, that's a good way to think of it. You know, the win-lose binary outcome would be they win the case and they get a ruling that what Google did was illegal, but we as consumers might not win if they don't do a good job of remedying the harm.