Hello. We're back, and hopefully you can hear us well. So, the last session of the day, Parallel. Should we just get straight to it? I think so. To my screen here, Parallel. Yeah, so Thor is the leader here. So why are we doing, like, what's parallel and why do we care about it? Right. Well, we care about it because of performance, which is not always great for Python. Sometimes running things on one processor is not enough. It takes too long. But as we'll come to, I think there are a number of things you should do before you actually try to parallelize your code. But parallelization is increasingly important.  You have eight or maybe more, and then you have all these HPC high performance computing clusters around the world, which have tens or hundreds or millions, hundreds of thousands of processors or even millions. Yeah. So how much do you expect or should one of the watchers of this course know? We'll say at the end of the course. Well, I think everyone should, the take a message will be that there are these options for parallelizing Python. It's good to know about them. You might need to use it sooner or later. We will of course not have any time to delve into any details. But it's good to know what's there, what you can do. You can come back and learn more in the future if you might need it. Okay. Yeah, that sounds like a good thing. So what do we need to know about the modes of parallelism? Yeah. So there are different modes, but I think there are, we should first focus on these five steps, which one should consider when something is taking too long. So have you noticed sometimes, Richard, that you're running something and it just takes too long. Like you don't have, the data set is too big or the computer keeps churning and you don't get the result. Back when I did more stuff with data, yeah, sometimes. Yeah. So the first step is what? Well, measure. So it's a very common mistake to start parallelizing, to start optimizing before you actually measure. And there is this famous quote by a famous computer scientist. How is it now? Premature optimization is the root of all evil. Right. Yeah, that's it. It's a bit drastic, but yeah. So the point of that is that you should first measure and there are these profiling tools in Python and in any other programming language where you get information on where time is being spent. And you would use that to actually see where the slow spots are. Maybe there's a function that gets called a million times. And that's where 90% of the time, total execution time is spent. Yeah. And that basically matches what I've seen. So my code's slow. It's usually not something radical like doing parallelation or something. It's realizing there's something that was just written quickly and doesn't make sense. And with some low effort changes, I can make it work well enough. Yes, exactly. So parallelization brings in another layer of complexity. So if you get adequate speed up just from looking at that bottleneck and improving it. I mean, it can be about changing the algorithm. Maybe you don't need to use a for loop. You can maybe use broadcasting in NumPy. I mean, bring in these really fast libraries like NumPy and Pandas and so on. And if that's still not enough, there are these packages for pre-compiling. Like you add some extra decorators and some extra code to actually make your Python functions get compiled ahead of time or just in time. So NumPy and Scython are two very well-known packages. So we're not looking into that here. But there are links at the bottom of this page with tutorials on Scython and NumPy. And after that, you can start parallelizing, right? Or thinking about it. So what are the main categories of parallelization we should think about? Yeah, I guess that's up to, I mean, there are sort of different definitions. But one clear distinction you can make is first the first category would be embarrassingly parallel. It sounds a bit negative, but what it refers to is simply if you're, you have to run some code a thousand times with a thousand different parameters. That's an embarrassingly parallel problem because you don't need any communication between those thousand runs. They can all run at the same time. And that's usually pretty easy to do. Yeah, and there are some tools to do that. Like one of the tools that are shown below. Dask will probably not cover it here, but you can look at it later. You can do embarrassingly parallel things there, but there are also like workflow managers and so on that enable you to automatize these tasks and run them maybe all at once on a cluster or something. And then beyond that, what comes next? Yeah, then there's the, I guess, somewhat more complicated approaches, multi threading on one hand. And on the other hand, multiprocessing slash message passing slash distributed computing. And the difference is that multi threading is when you have many threads, which will be running a different course CPU course. Share memory so you can have sort of like a loop or something with the different iterations of the loop are handled by a separate thread. But they're operating on the same on data in the same memory. While the lack the last category there multiprocessing or message passing. You can do that on a single computer with so called shared memory too, but you can also run it on different computers. Or on different servers, whatever. Yeah. So there's there's a question in HackMD is my code automatically going to use all CPUs on my machine. Oh, excellent question. And this actually leads us to something we wanted to show you. Yeah. So I'll talk about this big elephant in the room when it comes to Python and parallel. Yeah. So the global interpreter lock, what is it? What is it? So it's a design choice that developers of Python made a long time ago, that only one single thread in a Python process can actually compute can actually run Python code. And this sounds like it makes it just parallelism completely impossible. And this under the hood, technically this is called the global interpreter lock. Yeah. So we have a little demo here of it. So I guess the main point is that things like NumPy aren't bound to this because they're running stuff in C code. Exactly. So that's another reason you want to use these languages. So I have a quick demo. I would say I'll do this too fast. So don't necessarily try to do it yourself. But let's see. So here's my Jupiter. I'm making a new file. I'm going to make a new Python file. And I will copy and paste this into it. So this is something that uses NumPy. And remember that NumPy is written in C, mostly. Yes. So I'm saving it. Now I'm going to make a new terminal. Actually, I have my old terminal here. Python NumPy test. Should we do it like this first? Yeah. Sure. Okay. And I'll use this time built in. You don't need to. The code actually spits out the time. It doesn't. Okay. Then good. I need to save it. Hopefully. Okay. So it took five seconds. Five seconds. And while you're explaining that, I'll do the next part. Yeah. So there is. So there are some environment variables. That control. How many threads are you are being used. And this particular thing that Richard just wrote on num threads. This has to do with open MP. Okay. It doesn't make much of a difference on your machine. Because most of my processors are being used for streaming right now. That's probably it. So when I run this on my computer, a laptop, a Mac, it's four times slower. And that's because when you just run NumPy code without doing anything, it's automatically parallelizing it. It's using threads under the hood. So called open MP. Threads. Yeah. But I guess this is the main point here is this is a demonstration that NumPy actually is using multiple processors. Yeah. So. So that's the take home message here. So a lot of things come for free. But we'll also show you how you move beyond this and actually parallelize things yourself. Yeah. And next up is multiprocessing. Yeah. I think this is a go to library for many Python developers if they need to parallelize. So which paradigm would you say it uses? Yeah. So this is the message passing the multiprocessing paradigm. You can run multiprocessing on different servers on different nodes of a supercomputer. You can also run it just on a one machine. Yeah. And it's also, you can also classify it in a different way. So it uses this map reduce paradigm. That's where you map a problem over multiple processes. And then you reduce the results from the different processes into one. Yeah. Which looks like here. So we have a function that does some computing. It squares a number. Yeah. And we have here a list of input values. And the function. And map. So here we've squared each of these numbers. But here we've done it. Maybe you could do this, of course, with a for loop. Or I in 123456. Or this, but this is just a different way you can use this map function. Yeah. And once you have the map function, then you can do cool stuff. Like if we look down below here, we've got a multiprocessing pool. And the pool. Instead of using a built in map function to Python. We do. Multiprocessing pool that map. And it will detect how many processors I have. So you can use all of them and run all of these at the same time. Yeah. Or if you only have four processors, you know, you four will be computing one thing at the time and then the next batch will be shipped, you know, so it's, there's some automatic distribution of work to those pools of workers. So if you can't structure your code where you have some code. And variable input data. And then this can do things very easily. Do we go straight to the exercise then. Yeah, but there's this big caveat there. And the warning box, you have to scroll up a little. Yes. This might not work for you. It does not work for me. And according to the documentation of multiprocessing, this should not work. But we found out that it worked for you Richard. Yeah, which is. What doesn't work, or is perhaps not supposed to work is to run multiprocessing in an interactive environment. So Jupiter, or the Python command. You know, the Rappel or I Python. Yeah, or technically technical reasons. So what is supposed to be to work always is that you put your code into a file into a script, and you run the script. So that's the non interactive way, but clearly in some cases it could work interactively too. Yeah. But there is a workaround. So there is this other package called multiprocess, not multiprocessing, but multiprocess that one. And it's a fork, it's almost the same code is just modified a little bit, and you can install it with this pip install command. And instead of importing, you know, in the import statement there from from multiprocessing you do from multiprocess. That's the big caveat so you can try both if you're working in Jupiter, you can try first multiprocessing, it might crash. Yeah, or like it might not work you might give you an error. Which, and it's pretty cool how you've written it one way and by simply changing some imports you can change the way it's running. Yeah. Is. But okay, so how long should we give for the exercise. I think 15 minutes. But let's shall we describe the exercise a little bit. Yeah, okay. So one minute just to get everyone on the same page. So what do we do in here. So it's a toy example. We're computing pi. We have this function sample which takes in a number. And it checks if that number random number between zero and one, whether it's inside. No, it. Yeah, so it takes in a number of iterations. And then it computes random numbers x and y and checks if the sum of their squares are below one. Yeah, the unit circle. And then it increments a number so it's it's sort of a way to compute pi. Yeah, it's like putting random or putting random points on a square. How many of them are inside a circle on the square. And from that you can compute the pie. Yeah, and the idea here is that you can call it with a million and is equal to a million and you can run it on one processor. It might take some seconds. The task will be to use multiprocessing and this pool construct the the work. Yeah. And split up the work between different processors. Yeah. So sample takes an argument how many times to do it and we use multiprocessing to call samples say 10 times with 100,000 each or something like that. Okay. Yeah. And there's a solution as a hint, if you'd like. Okay, well, I guess we will send you to it. Yeah, and keep the questions coming in. Yes. Okay. Talk to you soon. Hello, we're back. All right. It seems that some of you had some issues. Yes. But I think they will be addressed in HackMD and in the document so it's understandable that it doesn't work for everyone. I mean things can can go wrong. Yeah. I hope it will be useful at least to have a look at the solutions to see how it's supposed to be done. Yeah. And this is like once you get to the parallel stuff debugging it can be quite tricky. I mean, this is we're not dealing with the parallel code itself, but yeah, don't get discouraged and keep working at it and see our examples. And if the examples don't work, let us know and we'll fix them. Yeah. But for the next half or 15 minutes. We have two other topics to go to. See, I'll switch back to my screen. So there's MPI. And I believe this is a demo. Yes, it's mostly a demo. If anyone really wants to type along, you can do that, but we cannot guarantee that everything will work because MPI. MPI, it was passed you to install MPI for PI was part of the software installation instructions for the environment. But that's like the most ever prone package among the packages that we are to install. Yeah. So there could be an issue, but let's see, you can try. Yeah. So MPI, the, the lesson says it's message passage passing interface. So what does that mean exactly. It's an old package, right? Like it's an old library. It's an old standard. I mean, been around for 40 years, maybe, or maybe more, I'm not sure. And it's still a standard workhorse of HPC it's used, it's installed and available on every single super computer on the planet. And it's a very different model from what you're used to. I mean, if you have written many serial programs, you just thought about, yeah, running things on one core. You really have to take a different perspective. You have to think differently when you use MPI. That's the bad news. The good news. It's, it's really powerful. You can do a lot of stuff. And you can paralyze a lot of different problems. And maybe let's just walk through those items quickly. So you talk about tasks or ranks. So that's the parallel processes that are being run at the same time. And they each have an index. They have a number, zero, one, two, three, and so on. And they manage their own memory. So this is a distributed memory paradigm. They communicate explicitly. So when you use multiprocessing, you use this parallel map, the PMAP function to automatically sort of send out work to some workers who will then communicate back their computed result. And a lot of stuff happens automatically there. With MPI you really have to do it manually.  And what the main sort of gotcha, the main thing that you have to keep in mind is that all the tasks also known as ranks, they run the entire program. So it's not like you saw with multiprocessing. There was only one line of code where you actually submitted jobs to the remote workers. Okay, it's the case instead is that, you know, every rank runs through the entire code and yeah, at the same time independently. And I think we have to look at an example to make this more concrete. Yeah. Should we try to scroll down. Yeah. Okay. So this is an MPI example. So, first is from MPI for pie import MPI, which I guess handles everything. It imports everything you need. And, and the code is familiar right is the same code you did for the multiprocessing. So yeah, okay so sample takes a number of arguments or number of iterations and returns to values. Okay, yeah. The rows that are marked in yellow are the MPI specific ones. The three first ones that come size rank they are absolutely standard you do it every time every single MPI code has this. First you have to. Well, the calm thing is the communicator. It's like the world in which your ranks live. So every rank every process that's inside a communicator can talk to each other. And then size and rank. You always can call this get size and the get rank methods to first the size is the number of ranks you have. And the rank is the rank of the process that is currently running this line.  So basically process zero can say, okay, I'm the leader. Yeah, typically. Yeah, usually you have one master rank master process that you will see how. Okay. So scrolling on down we see and tells us we do 10 million iterations. So if size is greater than one. So I guess that says if we're actually parallel at all. Yeah, yeah. Yeah, you'll often see conditional statements like this and MPI so if my rank is zero do this if my rank is something else to something else. In this case like if the size if we're running on more than one course, then split up this number and into the size. I mean you split. So yeah, you divide by the number of parallel processes. Yeah, otherwise you just and task becomes the value itself. Okay, and then we have this is about timing here. Well, timing and then it actually runs it with the number of tasks we need. Yeah. Yeah, and there that this is important to keep in mind so each rank will run this line independently of each other. And they will get their own value of this and inside circle variable. And will go into this function, the sample function. And, you know, generate the random numbers, everything that's inside that function and return a unique value of this and inside circle. Okay, yeah. For separate processes for different values of inside circle, and then they're collected or gathered an MPI terminology here. Yes. So this is one out of many methods that are common in MPI. This is the gather one. Maybe the most fundamental ones are the send and receive methods that's to explicitly like send a message between one ranked and another rank. But this gap and that's called point to point communication, like two ranks explicitly are communicating with each other. But this gather function here is it's called a global. It's belongs to the global communication methods. And in this case, what happens? What do you think happens, Richard? Well, an inside circle and root zero. Does this mean that if you're zero then or zero gets the values of everything else? Yeah. And does it add them together automatically? I don't see a sum here. No, it's not a reduction. There is another collective communication method called reduce, but this is just to gather the data. Okay, so this becomes a list. It becomes a list. Inside circle. Okay. Okay. Yeah. So down here we see so if it's rank equals zero, then we do some of that list. Yeah. And the final calculation formula. Yeah. Okay. And that's just demo this, right? Yeah. Shall we do that? Okay. So I'll be typing somewhat fast here. So maybe just watch. I copy all the code. I'll come back to Jupiter and make a new Python file. I will paste it all. I will rename it to PI test. I will remember to save it this time. I will go back to the terminal. And then to actually run it. What do I do? Yeah, you don't run it the normal way. You can run it the normal way. Python actually try doing that just for good measure. This works. Yeah. Okay. It's not crashing. So crashing, but it's running up. Yeah. Yeah. Which I guess it's good because then you don't have to use the MPI. No, exactly. Yeah. Okay. The MPI way. Yeah. The MPI way of running things is not to just run it like this, but to append at the beginning of the command. MPI exec. So MPI execute. And then you specify the number of racks, the number of. So let's try four. Okay. Yeah. And then Python. Python in parallel. Well, 1.1 seconds. So that was actually four times faster. Yeah. Four times faster. Unlike the multiprocessing example I did. Or no, I like the MPI example. Let's try. Let's have a look at the output. Yeah, you can run this too. Okay. Yeah. That was a bit faster. Okay. Yeah. So the output. So there's a print there before gather after gather. Yeah. So it shows all the ranks and there's something and inside circle becomes none. But for rank zero, then it got this list of everything. Yeah. So after the gather, the rank zero has all the results from all the other ranks. Yeah. And then computes the average and prints it. That wasn't too hard. No. Well, it's that these collective communication methods are. Sort of high level. It's many algorithms can sort of be mapped into that particular. I enter, you know, using community global. Collective communication. There's a lot. I mean, this is a whole science. I mean, there are week long workshops on MPI. Yeah. We learned there. So, yeah. But if it's interesting to you, if it you think this could help you in your project, whoever is watching this, then look closer into that. There are some links inside this lesson to further reading. Yeah. Okay. So what comes next? We've got a few minutes left. So. Yeah. Coupling to other languages. We mentioned in the library ecosystem some. So the basic idea is if you already have some code in another language, you can usually pretty easily use it from Python. And this is probably one of the big reasons why Python became so. A common in science. It was good for doing general work and good for connecting to things. So you could glue stuff together really well. Should we talk about desk and task use. Let's just mention it briefly. Again, this is another huge topic. You can do so many things with desk. Maybe some of you have heard about it before. You can do. You can do this embarrassingly parallel stuff. If you like, if you import task and you use task, you can sort of specify that. All these independent steps, you identify some independent steps should be run on different processes. But another very cool thing about task. Are these distributed data structures. So there's like an analogy to NumPy array called a desk array. Okay. There's an analogy. Yeah. You see it down there. Yeah. And also the same thing with a data frame. So there are these distributed data frames. Yeah, go ahead. If I have a data frame that say 100 million. Rows, I could use desk for that. And it's sort of automatically parallelizes it for me. Yeah, you can split your data frame over. Any number of computers, any number of ranks, not ranks tasks. Okay. Nice. That's specified down there, right? So the, with the array example. You're creating chunks, you're telling, you're creating a two by two random matrix. Okay. And then you're saying that you want to chunk up the matrix. So it's a 10,000 by 10,000 matrix. You create 1000 by 1000 chunks. And that's will actually put those chunks on separate CPUs, separate processors. And then you have all the other common NumPy stuff like transpose minus mean or whatever. Yeah. Okay. So is that the end? Task use, is there anything to say there or? This has to do with these embarrassingly parallel use case of desk. But I mean, it's such a big topic that people will just have to look at these links. There are links down there to other tutorials. Yeah. Okay. So that's the day. What was the summary of parallel? Oh, sorry. What is the summary? The summary of the parallel lesson. I guess three modes, profile first, use existing libraries. And if you need more, it's available even in Python. Yeah. Okay. But ask any questions on HackMD. Yeah. Is there any high level general questions or? Let's see. Or the technical integrity questions. We need feedback of the day. Oh, yeah. I think the day worked pretty well. Yeah. I think I hope everyone found it to be interesting and not too fast, not too slow. Yeah. Thank you a lot for these positive comments. Very nice to see. I'm a feeling one of the keys to a good workshop is don't let me teach too much. Unless I'm actually focusing on it. The more we have other people teaching, the better it is. I think your teaching has been awesome today. But today, but that today the day I haven't been teaching yesterday, not so much. Right. I dumped the panda stuff on you. I mean, it just was too busy. Yeah. But teaching in this collaborative way, I can just tell everyone listening that it's so much fun. Yeah. And it's efficient. Of course, takes more manpower, but less effort for every individual person. Yeah. Yeah. If you want to join us. So the good way is the code refinery chat. There's anyway lots of interesting discussion there about a scientific computing and technical topics. If you would like to teach with us, well, hang out there and let us know you're interested in being a co instructor. If you're in an organization that's not offering this suggests them to point, point us to them or point them to us and say, look, like if you advertise this to us, you can run your own breakout room. You can have a co instructor. This is an international partnership among. Is it only two countries this time? No three because separate here from Norway tomorrow. And we can take in other countries. I mean, there's no limit on how many people can watch Twitch. So, you know, the more people the better and we can even offer more of these other courses. Yeah. Maybe we should ask at the end of tomorrow. For what as people what they would like to learn next. I mean, is there any new lesson that people would want to see that's not already out there that is not being taught. Yeah. We're in a few weeks we're planning a course in February called a workflows course, where we'll go into more details about specific ways people put all of these tools together. Well, one person actually takes parallel stuff and array jobs and so on and puts it on a cluster. So if you're interested that might be a thing to contribute to. There's several more follow up courses we've been mentioning but the code refinery course is really important because it teaches sort of the software development side of stuff like the version control the automated testing and so on. Well, I have to use the opportunity actually to advertise the ENCCS workshop on Python. So it takes the next step after this one. So with a focus on high performance computing and high performance data analytics. What is it offered. Initially we were hoping to give it already in February but February is a bit packed so it will be moved to April. Should we do a live stream. I think we should opt for that. Okay. Well, that would be nice to do. Let's do that. Let's try so keep maybe I can even throw out some dates here. Yeah. We have tentative dates set. Anyway, the lesson follows the same format as what we're looking at now for this Python for scientific computing. We are thinking no actually may 18 19 May. Hmm. But yeah, block your block your calendars 18 19 May Python for HPC high performance data analytics. There's a lot more on desk there. There is more an MPI. There is number of sites on profiling GPU computing. Yeah. So tomorrow what do we have. There is so we aren't continuing parallel tomorrow because we finished it today. So yeah, I agree with the comment there's so much more to do but well, it's basically multiple week long courses. So tomorrow we're not so much about the Python programming itself but once you have the program how do you use it. So one is the dependencies. So how you keep all of these other things you're using organized. And then we talk about binder, which is a quick introduction to how you can allow this service called my binder for other people to run your code easily in the cloud. And then packaging, which is basically taking your code and making it where it's installable, which even for you, I've been happier once I've made stuff installable. And then there's a panel discussion. So we basically take all the instructors and anyone else who'd like to participate, and we come around and will answer any questions you may have. And you can see how much we agree or disagree and stuff like that. Then there's a quick outro and then I'll paste a zoom link into the Twitch chat and HackMD and anyone can join us and sort of talk about what we're doing together. Like, you know, you can talk interactively that way, if you'd like, sort of an after party. So be prepared for that and come with questions. And then will we do a similar course on R? I guess we don't have enough R experts to do this, but I'm sure there are. I can flag that there is a workshop being planned as a collaboration between several Swedish HPC centers to teach over three days, Python and HPC R and HPC Julia and HPC. HPC focused, but yeah, so it doesn't really perhaps exactly match what this person was hoping for, but I just thought I'd like that. Yeah. If you look around, there's countless courses on all these topics around. So as far as I know, we're the only massively open live stream course using our kind of strategies, but we hope there'll be more later. Yeah. So should we get going? Yeah. Yeah. So I guess we'll leave the feedback open for a bit more and see you tomorrow. Yeah. See you tomorrow. Thanks for today. Thanks a lot. Bye. Bye.


Thank you, Kate. Thank you, Tim. And thanks to the Linux Foundation for putting on such a great event. Yeah, so as Kate mentioned, Google made the decision to switch our embedded controller implementation from the homegrown RTOS over to Zephyr about two and a half years ago. And today I want to talk a little bit like how what Google's journey has been with Zephyr and how we found like collaboration has worked within the Zephyr community. So what am I hoping you're going to take away from today's talk? I hope you might be inspired to start contributing to Zephyr, or maybe you're going to start collaborating in a new area to you. And then also, you know, myself and my team at Google, we're just trying to make some new friends. You know, so Google is a large company, but even at Google scale, we can't invent and implement everything ourselves. So by switching over to Zephyr, we were able to leverage a lot of the great features and capabilities that are in Zephyr today. Device stream in particular, this reduced a lot of the custom C code that we had to create for every Chromebook model. And we also adopted Zephyr's test framework. And for our embedded controller application, we increased our unit test coverage of our application from only 30% lines of code to over 90% lines of code. And, you know, the community has been great. They've been very eager and willing to collaborate on new features. So let me walk you through kind of the story of one of our very first contributions with Zephyr. So very early on, we added a new fuel gauge driver into the Zephyr source tree. And we incorporated this into the existing sensor framework, much like all the other fuel gauge drivers. However, about a year later, I got a direct message on Discord that asked, you know, why didn't Google promote a battery API that we already use on Chromebook today and that you could see in our open source RTOS? And the short answer was, you know, this first driver we added was really kind of testing the waters with Zephyr. We wanted to see how easy or hard this effort community is to actually work with. We obviously like what we found. We'd like the community. And that was a big part of our decision to switch over to Zephyr. And if you're wondering, you know, who was it that asked if we could add this new API? You were all introduced to Enos yesterday. He was kind of number one contributor into Zephyr. I think it's close to 5,000 commits into Zephyr. So I shared Enos's request internally and we started scoping out the work for how we might upstream our battery API into Zephyr. We started with a fairly small PR focused just on battery fuel gauge, got some good feedback about how to structure our APIs, make it look more like other Zephyr APIs. And we also got some other requests, like maybe we should also add a charging chip API or we should also have a thread that helps manage all the stuff together. So we realized it would be good to kind of step back and sketch out and show what might a full implementation look like that's going to have a battery in a system. So we created a request for Comet RFC up on GitHub and it includes this block diagram here that includes kind of all this future work that we could be doing. So a power supply system plus these two new APIs for a battery fuel gauge and a charger chip. But just as important as sketching out what the end goal is, we also defined what the minimum viable product or capacity for this API could be for this effort. This allowed us and the reviewers to kind of focus just on the fuel gauge API that you see down here in the lower left. And any features or requests that were kind of out of scope of that, we could defer and just avoid the feature creep. So how does the scaling actually work within Zephyr? So Google created this RFC and this PR and we iterated on these until they got merged. So then at that point we could then start extending our own implementation up in Zephyr. But because this MVP is now in tree, the gate is open for other collaborators to start contributing. So in parallel, Intel made their own changes to the battery fuel gauge API. Then we had an individual contributor from the Zephyr community not affiliated with any company that added yet another battery fuel gauge driver. In this case, they're fitting it into this brand new framework that we're creating. Cirrus Logic made their own contribution. They created an RFC and a PR for the charging chip API. Microsoft joined the party with their own changes to the battery fuel gauge API. And then here just last week, I think, made their own contribution with another battery fuel gauge driver. And I should note the companies of Cirrus Logic and Microsoft, and I think this represented their very first contribution into the Zephyr project. So this instance really demonstrates kind of the power in the scaling that you get out of Zephyr. And having more contributors, more companies contribute results in a more feature rich and more flexible implementation. And for this particular case, it was great to see the diversity that we had. We had multiple large companies, a medium company, a small company, plus an individual contributor, all making contributions. Another thing I want to touch upon, another area of collaboration that Google is working on, is around sensors. So sensors have been kind of part of the DNA for Zephyr for a long, long time. And there's over 140 drivers in the tree right now. But there's been some limitations. There's been some asks to make improvements here. And so this is just a few issues that I was able to find up on in the Zephyr project now that are open. And if you even look at the last issue listed here, this was opened all the way back in 2017 asking, you know, can we make the sensors more performant? And this was even opened by ANOS as well. So about a year ago, Google and Intel started a collaboration work to improve sensors. So, you know, here's a simplified view of sensors today. An application talks directly to the sensor driver API. And it just fetches sensor readings one by one. So recently, Yuval at Google made a couple key additions to this driver API. So the first was an asynchronous read mode, which simplifies how applications can talk to these sensors. And then he also added a streaming API that allows you to fetch samples from high bandwidth sensors. And this also added time stamping into sensor readings. And then both of these built upon the great real-time I.O. services that was authored by Tom at Intel. And then in parallel, Intel has been working on adding a new sensing subsystem. And this includes some great features of adding time stamps to all readings. You can create virtual sensors that build upon multiple physical sensors.  And the very first PR for the sensing subsystem merged just last week. And this all builds on the great all the great sensor drivers that are in tree now. And, you know, make sure you check out the virtual talk from Hebrew and Ken. That goes into a lot more detail. And I think that actually had a pretty successful virtual office hours yesterday on this topic as well. So some other things, Google's working to upstream. We're working on upstreaming some ACPI power sequencing. This is going to manage all the power states of an application processor. And it's going to support x86 and ARM. And this is another collaboration effort that we're working on with Intel. And for those in Zephyr, you're probably familiar with Fabio. He was one of the release managers for Zephyr 3.2. He's been hard at work at adding an input subsystem into Zephyr. And hopefully you were able to check out his talk yesterday. And then on USB power delivery, Google has added and landed both the sync only and source only modes of power delivery. And some Googlers, Diana and Sam, put together a great virtual talk that walk you through the entire process of adding USB power delivery to your Zephyr application. All right, so, you know, as I mentioned at the top, we've been doing this for about two and a half years working on Zephyr.  But our partners, starting the beginning this year, actually are now shipping Chromebooks that are using Zephyr on the embedded controller. And this is just a sample of four devices that are on sale now that do include Zephyr as part of the implementation. And so where else is Google trying to use Zephyr? The first thing is Pigweed. Pigweed is a collection of open source libraries developed by a team at Google. And it really targets doing better and faster firmware development. And we're in the process of trying to get Pigweed adopted as an official Zephyr module so that Zephyr application developers can use Zephyr as a way to use it as a way to use it. And we know that Zephyr application developers can pick and choose the components that are going to work in their system. We have a couple virtual talks from Al and Yuval that talk about some of the integration you can do between Pigweed and Zephyr. And then a lot of Chromebooks also have a fingerprint sensor. We have recently started work to port our fingerprint firmware over to Zephyr as well. And this really helps future-proof our fingerprint design, making it a lot easier to switch to new space of supply chain problems. And we also have a couple hardware debug tools that are based off Zephyr. USB Type-C ports, reliable operation of those, is very critical to the user experience in Chromebooks. So we developed our own power delivery, USB power delivery analyzer, internally we call Twinkie. The latest version of this, Twinkie v2, is based off Zephyr and we're in the process of adding the board support files for Twinkie into the Zephyr tree. And then finally we have the chameleon board. This is an emulator for testing external displays with Chromebooks. And we do a lot of compatibility testing with this. Paul from Google had a great talk that goes into a lot of detail. That talk happened on Tuesday. But if you want to learn more about the chameleon be sure to just seek out and find Paul here at ZDS. And then we have, you know, I mentioned some of these before, but we had some other Googlers that were not able to travel to Prague. But be sure to look up, you know, Yuval, Al, Diana and Sam either up on GitHub or Discord if you have questions about their talks or maybe if you want to contribute. And in terms of in-person sessions, it's day three so the only session left that you can attend live is Aaron's talk on peripheral emulation, but I do encourage you to check that out. And then how to help. So, you know, where is Google trying to make some new friends? Sensors. You know, the work we've been doing here with Intel, it's going really well. But Intel and Google our use cases are really focused on laptops. So we'd like some other people from the community to start helping out here. I'm sure there's some other great use cases and applications that have some good ideas to make the sensors even better. And on the input subsystem, Fabio's kind of been a one-man show here. So if we could get some help, you know, maybe one thing is adding Z-Bus into this part of the implementation. And then finally for testing, you know, this is something that I think everybody in the project needs to help out on. Testing on real hardware is very expensive. And I don't have your hardware to test with, and you don't have mine. So introducing regressions can happen pretty easily. And, you know, within the embedded space, test is probably thought of as maybe a four-letter word. You know, let's get rid of that. And so, you know, I think a great goal, you know, a really big, hairy audacious goal for Zephyr would be that we would have emulators and tests for every single driver within Zephyr. So that's all I've got. Thank you very much and have a great day.


Everybody forced here rip in 2022 flawless football hobby two box full case number 35 This is a pick your team case. Give me just the second here read off names and teams Appreciate you all for filling this one up Ashams Steelers Anthony G. Dolphins Brad and chiefs Brandon J. Texans Rams and Giants Brett G. Cardinals Brett W. Eagles Caleb Broncos Dan F. Bills Frank with the bangles Gregory N with the Bucks Gregory J with the Browns Gregory M The Falcons Lions and Titans Hans H. Packers Harper Washington Heath F. Cowboys Jeffrey H. Jets John D. Vikings Jonathan L. Chargers Michael S. Panthers Nathan J. Patriots Patrick R. Seahawks Philip N. Colts Sean M. Niners Stefan L. Jags Stephen A. Saints Theodore Ravens and Raiders and Tom B with the Bears Let's get to it everybody Get to it. So tomorrow I'll be training rich on how to break Guys want to check that out? So Brandon Totally cool man appreciate it. What's up, Brandon? Jumbo patch 20 of 20 Alec Pierce rookie for the Colts Colts now Next up that's a cool 125 dual patch Chargers. You got Eckler and LT to do it all running backs right there Player warren Eckler non-specific LT there you go charge to 25 Giants with that sapphire color match blue there Harry Carson patch auto 2 out of 5 Here go New York Giants Sweethead Cowboys now patch auto 2 of 15 Ezekiel Elliott Hey Zeke good to see him signing in products this year. That's from 2021 flawless Zeke to 15 good stuff Dallas. They're one of the last teams to Rookie frame signatures Garrett Wilson 11 of 15 for the Jets Heck yeah Jets nice hit good interbox here Jeffrey H with the Jets rookie frame signatures 20 of 20 Lions Jameson Williams One of the last teams taken there you go Greg Jamo auto 20 of 20 How about keep it in the Motor City with a Barry Sanders auto 7 of 15 Super Bowl MVP auto Falls achievements There we go last spot. Hey, I put him on sale big time Greg's one that took him Greg gets a sweet Jamo and Barry Sanders very good interbox very good interbox of that briefcase Sick Barry Donovan Mitchell signing in that's sick Vaughan. That's awesome, man Were they 399 Greg? How much did the lion? How do we sell the lion? They were they were cheap big time on sale The Lions, yeah 399 for the Lions Sapphire number to 15 is Joe Montana 100 bucks they great not very not pay 100 bucks for Montana to 15 for the Niners on the sapphire gem hatch card Ravens Lamar Jackson 18 of 20 nice for color Peek that white at the bottom left there They're Ravens good stuff Diamond guarantee he goes with the Montana Hey Hey, well another one of the last teams taken here dual patch rookie auto Jamar chase 13 of 15 Was that Frank with the bangles okay? Yeah, Frank Congrats dual Patrick you know the 15 Orges pretty good box overall. Yeah, man. A lot of teams do well We like that for group rate purposes a lot of good hits overall. All right next one. I'll have to play for a bag next week Yeah, who is the who's the bangles back up again? I didn't recognize the name when I saw it last night They're probably trade for Josh Dobbs Which ones you get theodore? Oh Ravens the Lamar patch.  We don't sell the briefcases theodore We only send them out for if somebody's a personal or if there's like a damaged card. Yeah, I mean Yeah Close schools and Broward You have like a couple days ago, too. Yeah Dual Jersey to 15 is Earl Thomas Seattle a game warrant for Seattle Rookie showcase quad patch to five is breeze Hall for the Jets Nice emerald color match their jets Ravens another nice one dude. You've got a Justin Tucker auto now all pro ink to 15 That's theodore again. You don't are Russell Westbrook has autos too. Are you sure von? Are you sure that Mitchell and Russ signed? That's huge if they've if they've signed That's big and they got Tim Duncan, too Okay, Penini. I see you with basketball. He needs to step up basketball for the lose that license, man And to 15 Ravens patch auto is an quant bolden Holy cow Ravens. That's a disgusting an quant bolden game warrant patch Love the an quant he's doing it up, man. Grab the Ravens for cheap, too You want how much theodore got the Ravens for theodore got the Ravens for? I'm marketing out a few times 169 169 Score star swatch signatures the 10 Kirk Cousins, Minnesota cool hot cool card, Minnesota Vikings like that Kirk Probably the cleanest auto in football currently Dude kicker love kicker love Patriots awesome hit here four of ten jersey number a three-time champion auto Adam Vinitieri That is nasty. Oh my god That's a that's a big car dude the Tucker and the Vinitieri do not sleep those guys sell That is sick man three-time champion auto jersey number to ten Adam Vinitieri Yep, that's a big one right there. That's a big kicker hit That's going in a mag. That's a mag card right there holy cow and Then a Kenneth Walker the third rookie out of the 25 first seat at all great hit Pat's might have been chasing some Zappy since I think he's gonna start right Hey, not bad to get that Vinitieri. He'll always sell a last three cards of the break Sapphire jam out of 15 is AJ Brown for the birds Diamond guarantee there best receiver maybe best receiver right now, maybe hot five at least dual patch sincey burrow and chase Two encased cards for the bangles in this one 17 of 25 burrow and chase dual patch go Cincinnati and a Penmanship auto three of five Josh Allen with that color match in 2021 flawless Buffalo sick hit That a sapphire color match Joshie He's cut down at the turnovers but that It's great man. There we go. Bill's awesome hit He mentioned a lot of vets in that one. There were some good rookie hits though The Kenneth Walker the breeze Wilson Yeah, good for group rate though I won't recap as we got time to do it Relics here of Breeze Hall Earl Thomas on the game used Echler and LT Alec Pierce autos a Kenneth Walker the third curco change to ten and Guamboldon patch auto Justin Tucker auto Jamo Zeke patch auto Harry Carson patch auto Bangles dual patch was burrow and chase AJ Brown sapphire patch card Lamar for the Ravens a 20 Sapphire of Montana to 15 Josh Allen auto to five for Buffalo Barry Sanders auto to 15 for Detroit Wilson rookie auto to 15 for the Jets Benetieri auto four of ten for the Patriots loved that one especially with the Three-time champ auto. I know he was a four-timer was Patriots card and tool patch rookie auto to 15 jimar chase for the bangles That does it that does it. Thanks again, everybody. I've got personals and or origins coming up


So who am I? I'm Vinayak Mehta from Bangalore, India. I'm the author and maintainer of Camelot Index Caliber, the two Python packages that this talk is about. During the day, I work as a data engineer at Groofers, where I build tools and services that help stakeholders in the system to take data-driven decisions for the business. And we deliver 25 million orders to 16 cities, and that number is growing every month. So, like, we are hiring across roles at both our offices in Gurgaon and Bangalore. So, like, if you want to know more about that, you can catch us at the poster presentation that is happening at 4.15. I'm also an organizer at PyData Bangalore. We just started out in June this year, and we are doing, like, monthly meetups. This month, our meetup is on October 19th. So, if you're, like, in and around Bangalore, just consider submitting a talk by opening an issue on the GitHub repo. So, what does this talk about? I'm here to talk about how you can extract tables from PDFs very easily. I believe each one of us has encountered a PDF at some point in our lives, like resumes or research papers. A PDF stands for the Portable Document Format. So, this is a high-level overview of the talk. I'll briefly go through the history of the format. I'll touch upon some basing problems I've faced while extracting tables from a PDF file, then demonstrate how you can use Camelot and Excalibur to do that, and then we'll finally discuss the roadmap of these projects and maybe do a Q&A if we have some time. And yeah, there'll be some Python fun facts.  So, let's begin with one. Why is Python called Python? Anyone? Yeah, correct. So, while he began implementing the language, Guido Van Rossum was also reading the published scripts from Monty Python's Holy Grail, which is a BBC comedy series from the 70s. He wanted a name that was short, unique and slightly mysterious, and thus we have Python.  So, like, what if it was called Monty Kangaroo's Flying Circus? Like, I wonder how history would have shaped. Like, we might have been at KangCon. Let's get back to the topic at hand. So, PDF was born out of the Camelot project almost 30 years ago. This is a six-page memo by Jay Warnock, the co-founder of Adobe, where he gives an outline of the Camelot project. He describes the problems people were facing at the time to visualize a material between different computer systems. So, the first line is kind of like a high-level summary of that paper. The goal was to make documents that looked the same on any operating system. You're using to look at them and should print the same on any printer, like, as the author intended. It was created out of a subset of PostScript, which was a page description language, which is a page description language, which had already solved this view and print-anyway problem. PSN itself is quite broad, like, a programming language in itself. In PDF, or a PDF is designed to be self-contained. It encapsulates components required to render a document on different systems, which include text, fonts, vector graphics, and raster images. All these components travel with the document wherever it goes. Some more history were the PDF. It was created in the early 90s by Adobe. It predates the worldwide web in HTML. It was a proprietary format initially, but was released as an ISO standard in 2006. So, at a very high level, a PDF contains instructions to place the components that I just mentioned that X, Y coordinates relative to the bottom-left corner of a page. So, think of the bottom-left corner as an origin on a 2D plane, which means that words are simulated by placing some characters closer together, and sentences are simulated by placing words relatively far apart. So, in this case, in Quick, Q and U would be, like, somewhat closer. And when the next word begins, K and B would be placed somewhat apart. So, how are tables simulated? By just placing words, like, in lines, like they would appear in a spreadsheet in reading order. Basically, they just look like tables. There's no information internally about whether a column is a column or a row is a row, or what relationships are there between cells. This drawback of PDF, having no internal representation of a table structure, makes it difficult to extract tables for analysis. Sadly, a lot of open data is released as PDFs, like, in millions of PDFs, possibly billions of PDFs. Like, a format that wasn't designed for tabular data in the first place. A better format to store tabular data is the CSV, which stores tabular data in plain text. Each line of the file is a table row, and each row consists of one or more columns separated by commas, and hence, comma-separated values, which is its full form. Or JSON. CSV and JSON files can be directly read into, like, an analysable table structure using, maybe, pandas or a lot of other open-source tools. Now, let's go back to tables inside PDFs. If you ever tried to copy-paste a table from a PDF, you might have found it that it's not very easy to do. Like, most of the times, you have to copy each cell one by one and paste it into a text editor or maybe Excel. In 2016, I was working on scraping open data from PDFs for a startup. These are some of the PDF tables that I worked with. Like, when an organisation wants to release open data, it comes up with a bizarre and colourful table format. There's no set standard. Try to imagine copy-pasting data from hundreds of different PDFs and with hundreds of pages, man, that's not scalable. But there should be a better way to get a table out of PDF, right, without copy-pasting each cell.   Tabular is the first one that I tried. It works really well. Sometimes it has a nice interface. It's Java-based. Then there's PDF Plumber, which is Python-based in open-source. PDF Tables, which was originally open-source, but now it's proprietary. PDF Table Extract, which is unfortunately no longer maintained. And there are various other free and paid online services. Small PDF is the one that I tried. Problems with existing tools. Let's take this PDF, for example. This is a weekly disease outbreak report released by the Ministry of Health and Family Welfare in the Government of India. It contains weekly data about the number of cases in deaths deported for various diseases in Indian districts. With a comment column on what action was taken to prevent that outbreak. Looks like a very easy table to extract, right? It has clearly defined lines in just seven rows and ten columns. Well, this is the output when you pass it through Tabular. You can see like the table headers are in different rows and the column is all over the place. This is the output of PDF tables. They have a website where you can upload your PDF and then download the extracted output. It works slightly better, but it costs money. When these tools fail, you're just left with a badly extracted table, which you now have to clean up, adding extra time between the data and its analysis. And there's no possibility to get even like 80 or 90% of the table out nicely by tweaking some knobs or some parameters. So like maybe your data cleaning time is shorter. One solution that I tried when these tools didn't work was PDF to text, which is pre-installed in most Linux systems under popular utils. This is a sample PDF table that I created using Latek. You can use PDF to text like this, PDF to text file name, and you can use the hyphen layout option, which will extract all the text from your PDF using and preserve the layout using white spaces. But it has its own set of problems. First of all, the output is a text file and you have to have a post extraction step where you have to make sense of the underlying data table, maybe using complex regular expressions, which is expensive and time-consuming. Imagine having a GitHub repository containing hundreds of scripts for different types of PDF tables. That is neither scalable nor maintainable. To overcome some of the problems of not having a tool that gives you more control over the table extraction process, by being configurable and developer friendly, I worked on developing and open sourcing Camelot at SocialCops. And I designed this logo, taking some inspiration from SciPy and AstroPy. You can see the snake around the table. Well, why Camelot? Because it works well out of the box, pretty much, for most cases. And it auto-detects the way the table is without you having to do anything. For complex one, there's some table extraction parameters that you can use. Like, for example, you can say that that's not a single column that you recognize. That's actually five different columns that these offsets. A feature that a lot of users like is visual debugging using Matplotlib, which helps you visualize all the components that the library found on the PDF page. And those visualizations can help you tweak these different options that a library provides to get a better output. It also exports to all the useful formats like CSV, JSON, Excel, HTML, and even Pandas data frames. So you can directly extract a table out of a PDF and use it in your data analysis workflows. And it's written in our favorite programming language. It's MIT licensed, and it has excellent documentation. Let's do a short demo here. So this is a Jupyter notebook, which I've written. Is it, can everyone view it? Can everyone view the code? Okay, I'll still make it larger. So all you have to do is import Camelot, then you do Camelot.readpdf. This is, it has an API that is similar to Pandas where you do read CSV or read HTML.  So the wrapper, they're like, it's showing you that it found one table on the PDF page. And this is the PDF page that I showed earlier. Oops, yeah, this one. Oh, it's also on the next slide. So you can access it in attribute using tables.in. You can access each table within a table list using the indices. So if you do a tables zero, it'll show the shape of that table, which is your seven rows and 10 columns that are actually present on the page. You can get a parsing report of how the extraction process was handled. So if it's a good accuracy, then your table was found, like, your table would have was extracted nicely. Oops, let me scroll down. Then you can access the table data frame using a table zero.df. And it's the same table that was found on the PDF page, like this. After that, you can export your whole table list into CSVs. So if you do an export table.csv, and you'll see that since one table was found on only one page, it exported one CSV, then you can plot all the components that were found on a PDF page. For example, the text or the grid that the library detected. The table boundaries that were detected. The lines that were detected. And the line intersections that were detected. So like maybe if you didn't see enough intersections, you could tweak some parameters to get more intersections, which would mean, which would signify that your table was extracted nicely and it was recognized nicely. Cool, let's go back to the demo. Oh, the presentation. Oh, wait. So also the documentation is on read the docs. So if you go to the advanced usage section, you can see all the parameters that the library gives you. And like all of these parameters have illustrated examples so that it's easy for you to understand. Cool. So this is a slider added, just in case the demo didn't work. Camelot also comes with a command line interface, which you can see by doing Camelot-ifnl on your shell. The easiest way to install Camelot is using Konda, where you do just a Konda install Camelot-py and you specify the channel, which is KondaForge. Using PIP, you'll have to first install the dependencies which are TK and GoScript. And then you can simply do a PIP, install Camelot-py, CV. CV because that is the most basic flavor that you want. It also installs OpenCV on your system, which is used to recognize lines on a page. So how it works. It is built on top of PDFminer, which is an awesome Python library that gives you all the components from a PDF page and they coordinate. There are two parsing flavors, Lattice and Stream, which the names of which were inspired from Tabula. Lattice looks for lines on a page by first converting the page into an image using GoScript and then using OpenCV to identify the lines on that image. And Stream looks for white spaces and the text alignment, for example, left, right and center to guess columns on a page. And there's a disclaimer, the library only works with text-based PDFs right now and not scanned images and documents. So fun facts ahead. As you can already guess, like you must be wondering why it's called Camelot. So Camelot is a castle in Monty Python in the Holy Grail and the Arthurian legend depicted in the film. Oh, wait. And another fun fact, the PyPI was initially called the cheese shop based on the Monty Python cheese shop sketch. But let's get back to the presentation. But what if you don't want to write code? Camelot comes with a web interface called Excalibur. Oh, wait. You can use a web interface. You just do Excalibur web server on your terminal, which I'll just do. Cool, it's running. We'll go to localhost 5000. So here you can upload your PDF. It's upload the first PDF. You can specify the page numbers that you want to parse. By default, it'll take the first page. And Excalibur is async by design. So you'll have to wait for the page to appear. Now you can auto-detect tables that the library recognized, or you can select the flavor that you want. In this case, we want lattice because it's a table with lines. And we just click on view and download data. Again, you'll have to refresh because it starts a background job that parses your PDF page. So we can see the PDF page was parsed nicely. And now you can download it in any format that you want. By Excalibur, it's a web interface, so it offers assisted extraction. It's easier than a Python library. Since it is installed on your machine, your data stays on your machine. It never goes out. And it's configurable with Celery for parallel workloads. Again, you can install Excalibur using PIP after installing the dependencies which are TK and Go script. Another fun fact sign. That's the last one, I swear. Or is it? You must be wondering why it's called Excalibur. It's named after the legendary sword of King Arthur. Another fun fact, the metasynaptic variables in the Python documentation are called spam and eggs instead of the traditional foo and bar because of the Monty Python spam sketch. You should check on Monty Python's flying circus if you haven't already. Cool, that was most of it. Now, this is a roadmap for these projects. So a lot of users seem to face issues where they can't install Go script on their systems because of different operating systems. So the plan is to remove Go script and open CV altogether. Then there are some performance enhancements that can be done to extract a PDF that have hundreds of pages. Then since you looked at the web interface, you can guess that it's very functional right now. It can be made more nice and beautiful. And we need to add OCR support to get tables out of scanned images and documents. And maybe your favorite feature. If you use these packages, then we should talk afterwards about how you use them. You can find these packages at these GitHub repos. Again, if you use them, I would really appreciate if you would donate your time by contributing back to these repos. And so we are also doing a camelot and Excalibur dispense on 14th and 15th.  There's also a hacktoberfest going on. So if you open forth pull request on any open source repo, you will get a T-shirt. And I won't give that T-shirt, but the companies that are organizing the T-shirt will. And you can find the slides afterwards at these links. And I'll be happy to answer some questions now. Thank you. Is this on? Yeah. Thanks. That was an excellent presentation. We actually do a lot of that. And we face... You're not very audible. Hello? Yeah. So we face similar problems. We actually do a lot of that. So I'm going to go ahead to my room and try this out. Couple thoughts. What if the tables are non-regular? Like one has the first row has four columns, the second one has two columns. Like you're doing a call span. You mean the cells are spanning across multiple columns? Yes. If there are lines on that table, then the library will recognize it very nicely. And you will be able to... Like the library would recognize the spanning cells and put the data in such a way that you can copy it over those spanning cells so that it's easy for you. Okay. That makes sense. Second question, if I may, is somewhat of your thoughts on OCR because OCR will get the text. That's quite possible now. English text, please. But not the tables, I don't think. Not the tables. As in the lines. Yeah, the lines... Like if we add OCR support, we'll still get the lines out using half-trans... Morphological transforms from OpenCV. But we'll do OCR to get the text out, actually. And then, like, assign that text in different table cells that were recognized on a page. Okay. Have you done any work on that? Maybe we should take this offline. Oh, yeah. We should totally take that off. Yeah. One question here. So, do you know any limitations today? You know, like, I know there are different type of tables that we really extract about this PDF format. So, are there any known limitations at this point of time which is not going to work? You mean the limitations that I've seen? Already, you know. Yeah. So, like, most of the time, if your PDF is text-based in the two-unicode map inside your PDF is correct, the library should be able to get your data out using some parameters or without using them. But there are cases where the encoding inside a PDF is broken. So, you might actually see phones that look like English, but it actually has garbage inside it because the map is incorrect. And then again, scan documents are, again, a limitation right now. Any more questions? I do have one question. So, you mentioned distributed workloads using Celery. What was that for? I'm here. Oh, yeah. Hi. Yeah. Just wanted to understand a little bit more about the distributed workloads point that you made. What does it mean in this context? Okay. So, by default, it uses multi-processing to, like, start those async jobs if you're not using, like, Celery. But there's something called an Excalibur.cfg, which is a file that you can modify with maybe the URL to your RabbitMQ queue. And then you can start an Excalibur web server and an Excalibur worker that will make it distributed. So, a single PDF you're trying to distribute and process it. So, each page will become a job, a next-action job. Okay, run in parallel. Yeah. And what was the plan for moving away from GoScript and OpenCV? What's the tentative? Yeah, so that is still a very shady area. Like, we still need to, like, talk more about how to do that. Okay. Thanks. Thanks. Hello. Sorry, here. So, yeah. Good evening. Yeah. I mean, I've just done a project right now, like, a few months ago. I was passing a resume project. That was a parser project. So, in that, the other formats were also there, like, RTF or document format or anything of that stuff. So, we were able to extract the data from those kind of formats. But when it came to PDF, it was very difficult for us to, you know, extract data from a PDF file. So, you mentioned in your slide, right, that PDF minor, you know, library. So, we had installed that, it was like kind of, it was showing some versioning problem because of which we could not actually utilize the proper, you know, thing of that PDF minor. So, can you give us some solution, like, how can we be done? So, if you're using Python 3, you will have to install PDFminor.6, which is Python 3 compatible. The earlier PDFminor has stopped its development and is, I think, can only be used in Python 2. Yeah. Maybe we can take this offline. I have another question. So, when you're giving the report after extracting your table, how are you exactly calculating the accuracy? Okay. So, PDFminor would give you the text strings that were found on the page used in their bounding boxes. And then, Camelot will recognize the table grid and while assigning those text strings into each cell, it'll look into how much that text box overlaps with the boundary of that cell. So, if the more it lies out, the less the accuracy will be. And the more it lies inside the cell, the more your accuracy. Yeah, hi, over here. So, you mentioned at the beginning of your talk that one of the motivations for you to make this tool was that otherwise you would have... You're not audible. Is it better now? Yeah. So, I said that at the beginning of your talk, you mentioned that your motivation for building Camelot was that otherwise your extraction pipeline would be very convoluted in too many steps.  But I don't always have to extract tables. So, does Camelot support non-table related extraction as well or what would be your recommended workflow for when there will be tables but not always? I didn't get the last part. What does it... I will have to extract tables from PDFs but not on all pages. So, there'll be a lot of pages where I only need texts and images. So, does Camelot support that out of box or do you expect it to be run with another tool? If I got the question right, you're asking if there are images on different pages, then does Camelot recognize those images? My question is if Camelot only does tables or it does text recognition and on top of that, it detects tables and text. So, mostly it does tables but you can also use it to extract paragraphs and other types of things. But a PDF miner would be a better library to use if you're not extracting tables. Cool. So, I had two questions basically. The first one is using Camelot, you are basically using the lines to make the tables, right? So, what if a table doesn't contain lines at all, just a table without, say, lines in the middle? Yeah. How would you classify it using Camelot in that case? Second question being about the two flavors you mentioned, lattice and stream. I'd like some clarification on what kind of differences they have and how to use them. So, for the first question, if the table is not constructed using lines and it's simulated using spaces, then you'll have to use a stream flavor which implements the numinance algorithm which is like a paper from the 2000s and it basically tries to guess a table structure on a PDF page which doesn't have lines. And like I mentioned earlier, lattice should be used when there are lines on the table and stream should be used when a table is simulated using spaces. I think that should be all of it. We can totally discuss this afterwards. You can catch me later. I'm also, by the way, I'm also doing PyData Bangalore post representation in 4.15 p.m. So, if you want more information about that, then you can catch me there too. Thank you.


Music and entertainment opens the program in the 6-8 temple with a medley of old favorites. When a tire wears smooth in a hurry or fails to give good mileage, it's due to the lack of knowledge on the part of the man who sold it. The general tire dealer saved his customers many dollars and a lot of worry because he is a factory trained expert. Attending these factory schools regularly, they keep posted on the newest and latest tire development. There is no guesswork in the general tire dealer's service. He knows the right tire to put on and the correct air pressure to carry. When you put this factory trained knowledge behind the tire of general high quality, you have a combination that means the longest, safest, and most economical mileage you can buy. And now for that funny comedian, Jack Benny. Thank you. Hello some more. This is Jack Benny, your Hollywood reporter coming to you with all the late news reports with the courtesy of the Hollywood close-up. These all knows all and brings out your defects. Let's go. Beverly Hills, California. The Bing Crossbees are proud parents of twins. They now have three boys and are the March Brothers worries. Hey, that's a good laugh to start with, isn't it? The Little Crossbees were born in the blue of the night. And Bing was seen giving everybody a coroner's cigar. It's the best I can do with that. Ocean Park, California. There's a rumor that Guy Kibbey was seen here going around with Joan Blondel. But don't worry, folks. It was a merry-go-round. Each had a different horse. Are you thrilled? San Goldwyn, California. Eddie Cander, famous stage and screen comedian, gave a big dinner at his home last night. Who was that, Jack? Just died in the girls. That was enough. There is no truth in the report that Caviar was served at that dinner. It was just freckles on the waiter's hands. That's what fool I was saying. Who is this California? What great screen star was seen on what night with what director in whose car? And what did he say when she said, little man, what now? P.S., who cares? Merry-go-round. Director claims he is so good, he has a chance of stealing the picture. Hollywood, California. Brian Parker has as much chance of stealing the picture as K.T. in second base. Yeah, Mary, everybody has the reason. Go ahead. Beverly Hills, California. There's a rumor that Gary Cooper, famous screen star, is that way about Mary Livingston? Is that the truth, Mary? No, but I hope he gives me an. Nice girl. She's cute, huh? Nice girl.  General Fire gave a blowout last night, and she has blowout proof, non-skidding, and has the popular low price of construction. Well, you can't stop that. You got one to read, Jimmy? Yeah. There is a rumor that Jimmy Griller and his orchestra will now play My Middle Name is Love. That's a beautiful play, Jimmy. That was Jimmy Griller and the boys playing My Middle Name is Love. Well, guy that Griller.  And we went 50-50 on everything. I bought the dinner. He paid the car fare. I furnished the cigars. He had the matches. He took me to a show. I bought the tickets. He sure meets you halfway. Wait a minute, Jack. Several times I reached the check, and you just happened to beat me to it. Well, reach a little further next time, Jimmy. You know what I mean? Get your money out in two-four times instead of the old-fashioned walls, you know? But outside of that, you're a nice fellow. Really, folks, Jimmy would give you his shirt. That is if you were a China man you wouldn't charge too much, you know? Say, Jimmy, we've been together now for eight weeks. The next Friday night is my last program from Hollywood, so I think you ought to say a few words to our audience.  All right. Ladies and gentlemen, I want to say that in all the years I've... That's fine, Jimmy, and a great thought, too, believe me. I mean, how many leaders can talk like that? You know, folks, a lot of people have asked me in fan mail and in person what Jimmy Greer looks like, and I think it is only fair to describe him to you. Jimmy is tall, rather handsome, and when you first meet him, you don't like him at all. But after you're with him a while, he's no bargain either. In appearance, I would say that Jimmy is the Terry Grant side. He looks more like General Graham. And speaking of General, it's the best tire on the market today. Oh, Jack, I read a great book last night. It takes you a few years. Isn't that a little over your head, Mary?  Oh, all right. Answer the phone, Mary. You answer it. Hello? Hello. All right. Do I go to work tonight? Who is this? Oh, yes, our murder mystery. Life at home? I'll see. Oh, many. Stay at home. Come over right away, both of you, and go to work. I need you. Goodbye. And now, ladies and gentlemen, continuing with our mystery, the Sue's murder case, or Who Killed Mr. Sue.  But to you who have foolishly neglected it by attending to your business, I will describe what has gone on so far. Mr. J. Gasilto, too, of the famous radio suges was murdered on the night of June 15 in his villa on the Rue d'Ille, which strangely reminds us of this mystery. And the only clue are our six shots found in a bottle in Mr. Sue's hip pocket and a jar of goldfish that won't talk. Say, Mary, have you any clues? No, just a run in my stocking. Now, what was the motive behind this murder? It was not homicide. It was not suicide. Maybe it was herpeside.  We will continue this mystery right after the next number, which will be sung by Philo Parker, the thin man. Play, Officer Donovan. I have an ear for music and I have an eye for the may. I like the pretty girlies with each pretty spoon at play. They go to deserts like sunny weather. Those with a munch of a mare. The pretty girl is just like a prey to my heart. I like the pretty girl. Mr. G's, boy, that was Frank Pinkerton singing A Pretty Girl Is Like A Melody from the Ziegfield Folly. And now for that battle of wits with the underworld. Good tune. Does Frank pay? No, but Queen does his 14th sense of honor. The scene takes place in Sergeant O'Hare's office in a 110th precinct police station. Or maybe I was transferred by now. I don't know. Curtin, play, Mr. G. The John Who? Yes, Sard. Where's Officer Clancy? The judge gave him 10 days for vagrancy. Well, get him out. After all, he's a policeman. Okay, give me the keys. Fine, locking up our own people. No money in that. Oh, Clancy. Yeah? Come out now, Cook. Thanks, Mary. Thanks. Hello, Sard. Clancy, what are we in for? Oh, nothing. I was walking my beat for two weeks and nothing happened, so I was arrested for loitering. Well, keep out of those neighborhoods. Clancy, we received thousands of letters from indignant taxpayers who are fervently looking for a conviction in the suit murder case. What do you mean, fervently? I don't know. That's what's worrying me. Now, listen, Clancy. I want you to bring in every crook in town. Get going. Okay, Sard. I need some help. All right. How many patrol wagons have we? Four. Take three and leave me one. I'm going up seven tonight. Now, get going. Answer that, Miss Johnny Who. Hello. 110th precinct police station. Yes. She did. What time does it happen? She don't say. Does she feel rare? Yes? No. All right. You bet I will. Goodbye. What was that? I don't know. I guess I had the wrong number. I'll take that next phone. Hello. Hello, Sergeant O'Hare speaking. Who? O'Hare. O'Hare. I never do get that name. O'Hare. What is it? A bald-headed man hasn't got. Sandro. All right, then. This is Dan Drough talking. What do you want? A man was murdered here seven weeks ago. Now, what are you going to do about it? Don't be so impatient. We've been working our fingers to the bone. Oh, shooting craft, eh?  Listen, what are you going to do about this dead man? Well, as Hawkshaw once said, leave it to me. And as Shakespeare once said, shell fruit to be. Goodbye. Goodbye. Goodbye. Why, this is an outrage. They can't do this to me. I'll call up the commissioners once. I'll see the commissioner about this. Come on, come on. Get inside. Get inside. All of you. What's this, fancy? I cleaned up every free lunch counter, pool room, and park bench in town. They're all for Spaces Characters. It looks like the gang have been trying to round up for weeks. Look at those faces. Which one of you is the leader? I am. And this is my orchestra. And what's your name? Garface Greer. Ray. I think I've been named, fancy. I don't know what it was. It's a commission to get. Wait a minute. I've got a hunch. Nobody leave this room. Ms. Donahue. Yes, sir. Get me Oxford 3131. Okay. Operator, get me Gladstone 9473. I asked for Oxford 3131. Well, this is the best way to get it. Hello. Here you are, babe. Hello. Hello, who is this? Police headquarters. Sergeant O'Hare speaking. Who? O'Hare, O'Hare. I'm not testing the name. O'Hare, what have you got on your head? My brother. We're acrobat. Wrong number. Goodbye. Oh, Sergeant. What is it, Clancy? What about this tough-looking egg over here? Maybe he knows something. He does look pretty suspicious of that. Where did you find him? Oh, hanging around the studio. Oh, yeah? Come here, you. What's your name? Bill Baker.  What do you know about this dude's murder case? Well, Jack, this is embarrassing. I was just sitting out front watching your program. Don't give me that stuff. What do you know about the dude's murder case? Nothing. And neither do you. Where were you on the night of June 15th at 8 o'clock? In Chicago at the armor plant. Well, you can buy the finest hams. The finest hams. Stop it, Clancy. Stop it. Why? Why? You can't mention Armour here. Why not? There are more features to the general tire than any other tire on the market. It's the low pressure that makes generals blow off proof and they can be purchased on convenient terms for the general tire dealer. I guess we're even, Baker. Not yet. I came up here to get that 10 bucks you borrowed from me at the David's Theater in Pittsburgh. Don't evade the issue. Why did you kill Mr. Sue? Don't evade the 10 bucks. Why didn't you write when you promised her? Nick Spill, your captain of the mystery. What mystery? I want my 10 bucks. He'll give it to you next week, still. He just paid off ready, Alan. Quiet, Miss Donahoe. Clancy, who's that other egg over there? No, he came in with Baker. Yeah, come here, you. What's your name? My name is Bottle, sir. Bottle, eh? Your name is Bottle. What's your first name? Nicholas Murray. Nicholas Murray Bottle, I see. You know who I am, don't you? Oh, yes, yes. Jack Wynn, the tire chief. Hey, Bottle, why didn't you save that one for our program? Quiet, sir. I'm not working for you now. Add a boy, Bottle. Now, wait a minute, you. My name is O'Hare, Sergeant O'Hare. I know you, Sergeant. What's that name, sir? O'Hare, O'Hare. What have you got in your head? Liquid Vaseline.  Ah, what is it? What are you laughing at, Bottle? What is it? It was funny when that gentleman over there said dandruff. Now listen, you two. I've had enough of this. My name is O'Hare. Get off the O'Hare. Who was that? That's Beetle, Jack. Oh, Beetle, eh? Oh, we're haunted here, too. Now listen, Beetle. Do you know who killed Mr. Toode? Yes, I do. Who did it? Mr. Toode was killed by operator 13 in the shadow. This crew will be continued next Friday night. But try and solve this mystery. Your guess is as good as ours. Play Beetle, Bottle or Jimmy. Stick around, Phil. We'll have some fun later on the program. You haven't got another bottle with you, have you? You got another bottle? Well, please, Jim. All right, play then. Which hour twins played by Jimmy Greer and his orchestra? Well, what do we do now, Phil? You know, you can't leave me flat now that you've butted in here. I don't care, Jack. It's great to see you again and talk over old time. There is. Remember the fun we had a few years ago in Kansas City when I was there playing the Opium Killer and you were laying off? Yeah. You remember in Chicago when it was vice versa? What vice versa? That was the product. That's right. But Phil, will you ever forget that theater we played together in St. Paul when we were on the same bill together? Oh, sure. Remember my name was in lights when you were kicking to the manager? Yup. Your name was in lights, all right. Yes, sir. And Jack, do you remember after your second joke, your name came down and they put Mickey Mouse up? Oh, yes. Hey, what's Mickey Mouse doing now, Phil? Oh, he's making another comedy picture. See, that's funny. Here we are, the three of us in Hollywood making pictures. Yup. Mickey Mouse and the two rats. Say, Jack, he used to play pretty good fiddle in those days. Are you still playing it? You mean violin. Well, it was fiddle in those days, yeah. Well, I still play it occasionally. Do you still use your accordion? Oh, once in a while. I bet the audience can't guess what they're going to do. Not much.  You know, for old time's sake. Sure, I'll be glad to. I'll close your bottle. Oh, sure. They've been rehearsing it all week. I'll reflect. I don't know. I generally have stuff with the audience, but any number you folks would like to hear. Anything to talk about? Humorette. Okay. Doesn't make any difference to what you call humorette. Is that it, Phil? Yeah. How does that go, humorette? Humorette? Wait a minute. What do they want? Humorette or Loved by Neighbor? Loved by Neighbor. Loved by Neighbor. Loved by Neighbor. How does that go? Let's play. One of the go Loved by Neighbor. Loved by Neighbor. Never mind. Rep Assessment, Loved by Neighbor. Da, da Da dear da dear da dear da. Loved by Neighbor. Loved by Neighbor. Got your pay and Loved by Neighbor. Phil, give me A on there disclosed A. Good, almost. Good. Thanks for the hard work. We have Iks. Okay. Humorette? visited. Ready? Surprise. You're seeing. Surprise? You are. Harmer's Hammett is the best. And General Tyer beats the rest. A-da-da-dee-da. Love what I'm doing. Is that it? I told you to take up the corner. Good hot jack. Don't you think it's awfully nice when you have a piece of ice? That was Phil Baker and Jack Benny playing humorous. And humorous to anybody about the general blowout to Tyer and you will find, you will find that... play. In every field it's a specialist who is best qualified to serve the public. The general Tyer dealer in your community is a factory trained Tyer expert. He owns his own business and his first concern is to see that you get the most for your Tyer dollars. If you've never done business with him, why not stop in his store tomorrow? Let him show you how important proper Tyer equipment is to your safety and comfort. He can tell you why one Tyer can wear out in half the time. He can show you how easy it is to correct many things that cause irregular wear, thriving fatigue, shimmy and unexpected blowouts. The general Tyer dealer is a real Tyer specialist. He can save you many dollars and assure you greater Tyer safety because of his factory trained knowledge. If you want greater safety, greater economy and greater mileage, the general Tyer dealer is the man to go to. This is the last number of the 16th program in the general Tyer series. Be sure and listen in next Friday night and find out who killed Mr. Seuss. Come on, Mary. I'll take you home now. Sorry, Jack. I'm going home with bottles. Where's Beeple? He went up to the country for the weekend to make noises. Oh, good night, folks. The general Tyer program starring Jack Benny has come to you from the NBC Studios in Hollywood, California. The tune, Shine on Your Shoes, is from the production Flying Colors. Don Wilson speaking. This is the National Broadcasting Company.


What's up everybody, Jason here for jaskiescasebricks.com, 2021 Pinini Flawless Baseball just sold out. This is a one box briefcase break, Randon Teens number 22 and of course just please know guys we got these loose boxes from our suppliers, I don't know if they came from the same case but I have two of them here, so maybe the supplier was selling them as loose boxes but it could still be from the same case or it could just be monsters in all these loose boxes we got. So again we can run back number 23 after this, but again this is number 22, here's a dice roll, here's a customer name, so from Joe down to Daniel Lewis, bought those spots straight up and then Ed down to Darryl, won those spots in an optic pack filler and then there are the teams, good luck guys, roll it, one and a four, five times, good luck, one, two, three, four and five, five times, Ryan at the top down to Eric, five times, one, two, three, four, five, five times, five times, Oakland A's down to the Pittsburgh Pirates. Alrighty, so Ryan with A's, Daniel with the twins, Kevin with the gardens, Daniel with the giants, Ed with the braids, Paul with the rays, Joe with the brewers, Daniel with the white socks, Red Sox, Gregory with the Rockies, Daniel with the Mets, Gregory with the Phillies, Daniel, you have the Yankees and the Blue Jays, very nice. Darryl with the Reds, Nancy with the Dodgers, Terry with the Padres, Daniel with the Cubs, Ed with the Cardinals, Darryl you have the Astros, Daniel with the Marlins, the Deepbacks, Darryl with the Tigers, sorry, Kevin with the Rangers, Daniel with the Orioles, Nancy with the Royals, Daniel with the Nationals, Brandon Hall with the Angels, Ed with the Mariners, and Eric with the Pirates. So if you guys do want to trade, let me know in the chat, try to trade amongst each other, or try to maybe propose a trade for a team that maybe you've interested in there. If you don't see anybody else chatting with you, then try to trade with each other there in the chat. All right, we're going to trade guys. So let's see, Mets for training, I'm interested, I want the Padres for the Mets, I would do that, Terry, confirm. Padres for Mets, gotcha. Padres for Mets. All right, let's close the thing guys. Padres, Daniel you're not going to have the Padres, and then Terry, you're going to have the Mets, correct? That's what we did. And let's see which briefcase we're going to do here. Like I said, we have two loose ones, top one, two, three, bottom four, five, six. Let's see what we can get here. Moved over here, and it was a one. So one, one, two, three. So the bottom one will be for the next break, and I'll be able to post it up as well guys. Remember these are all loose boxes though, they didn't come from the case. So see what we can get here guys.  Looks like Terry now has the Mets, and Daniel has the Padres. Good luck to both of you guys. All right, let's see what the fuss is about with this flawless, right? Everybody wanted to do it, finally we did it. Let's see what this loose box is going to bring us. Either happiness or some sadness. Hopefully happiness. All right, so there should be only two encased cards unless there's a cleat in the middle. Nothing in the middle. Looks like potentially the diamond is on the left side, hits on the right side. So it's empty there with just the keys, a thick box, right? Kind of a thickie. All right, here we go. First one is a Vladimir Guerrero for the Montreal Expo, which of course will be going to the Washington Nationals. So that'll be going to Daniel on the board buddy, nice patches right there as well. That is a five out of 10. We have a six of seven Duke Schneider, legends there for the Dodgers. LAD is going to Nancy. Nice one there Nancy. Solid hit right there. We have a Fergie Jenkins for Chicago. Cubs going to Daniel as well. Nice signature there as well. Awesome. This card just looks so good. We have a Fred Lynn for the California Angels. Angels going to Brandon Hall. Nice two of ten. And we got a button. How about a little button relic and autograph for Kyle Schwarber and the Chicago Cubs edition. There you go. Four of 10. Not alone for Daniel and the Cubs. Very, very nice. And we got a Jim Rice, one of 10 for Boston. Also for Daniel. Nice break for you Daniel. I know you're putting the work. Getting rewarded. Capsic Cleveland hit for me once today. Let's go. Well guess what? They hit for you again right? Eight of 10. Tristan Mackenzie. Very nice RPA there. Eight of 10. Basketball also only has two as well Rex. Baseball occasionally does have three though. But basketball only carries two as well. Football is such a bigger lead though you have to remember that Rex. A lot more players on the roster and you know 60, 70 plus teams and then plus Hall of Famers potentially. Obviously you're not featuring every single player on the team. But it's a bigger league per se. That's what I'm trying to say. Rockeroo. Five of 10 for the Minnesota Twins. That's going to Dano and the Twins. Very nice. Five of 10. Alrighty. Here we go. We'll go to from the left to the right. Diamond gem first. Rookie. Joe Adele. 15 of 20. Angels went to Brandon Hall. All right who do we got here? I know we did have a lot of veterans in this one right? Ooh looks like a catcher. Who's a catcher? And it's also a rookie. I think I know who it is. Joey Bart for the Giants. That's last spot Mojo Daniel. Six out of 15. Look at that. Nice patch in there as well. Six of 15. Joey Bart. Rookie RPA from Flawless Baseball. And again that is Daniel last spot Mojo. That's pretty nice. And just like that guys that was the break. Very quick right? So there you go. Two in case cards. Yeah we did get a lot of old school cards. There's Trism and Kenzie. Nice little button. Relics. Very nice. So again the second half again I don't know if it's from a sealed case. This way that it came from you know but they opened it and sold it to us that way. Or it's going to be a totally complete different box and potentially have some bigger hits than what we just seen here. So I will post up number 23. I'm going to only post up 10 spots planning to do two fillers but as we did for 22 if we sell out the 10 spots pretty quick I will always cancel the next filler that way we only have to do one. But for now I'm going to keep it at that we have to do two fillers okay. So two fillers and 10 spots. So appreciate you guys. Jaspy's casebricks.com