Good morning. Can I, is it too much to ask you all to kind of come a little closer? Just a little closer. 10 seconds, just yes. Thank you. If that's okay. Only if that's okay. You can also stay where you are. Thank you so much. Alright. So, we're on time now, but usually I am late. In this particular situation is 9.03 and I am late. Despite the fact that five minutes ago, no, eight minutes ago, everything started pinging, you know, Slack ping me, Google Calendar ping me. I got a text message, I got an email that I had a call. I got all the pop-ups about having an email about having a call. Yet I am late because I went to edit the agenda item and lost track of time. It doesn't really matter how benign the reason I'm late is. When I walk into, walk into Zoom, Jenny looks really pissed. What is happening, Petja? This is turning into a thing. You're late for every meeting. This is totally disrespectful of my time. It's turning into a real thing for me, she says. This is the first thing she says when I open Zoom. I take a deep breath and try my very best to not roll my eyes. And I mutter an apology immediately, instinctively. And then I try to move away, promising, semi-promising that I'll try to not be late for meetings anymore. Behind my forehead, my grumpy, moody Bulgarian is just losing it. Why? Why is this a team? Why is this such a big deal? Why is three minutes such a big deal? I stay composed, but I don't have time to process. Usually, when I get into a situation like that, you know, experience from the last several years has taught me, stop, Petja, count to ten in those ten seconds. You'll be able to rationalize and it will be okay. But I don't have time. I'm already late and we've got stuff to do. So I just stay pissed until the rest of the meeting. And I'm pretty sure that because of that, we probably get a little bit less done than we would have if I had taken ten seconds. The thing is, I realize once I'm off the call and before I start granting to somebody else about how, you know, Jenny is about being on time, I take these ten seconds and I, like, it always takes ten seconds and I got it. You keep forgetting Petja, I remember. Jenny is actually American. She's not French. You know, just because you met her through the French WordPress community, it doesn't matter if time is as fluid to her as it is in your mind. Jenny is American and you know, you know that about Americans. Time is money. That's what they get taught from a very early age. They're very particular about time. You, you're not particular about time. I was born in Communist Bulgaria, which was particular about things, including time, but the regime fell when I was seven and then it was chaos. In the 90s, which actually brought me up, there were no rules. And actually people rebelled against everything that was to be related to the regime.  The thing that I did to rebel because I was a very good girl that went to school and had good grades. I mean, the fun fact is I almost didn't graduate because in 12th grade they almost threw me out of school because I was notoriously late for class.   I've always been able to kind of get away with it, you know.   But I never really thought of time as being such a big deal, a particular time in general. But it is. It's one of the eight big things that actually differentiate between cultures to an extent so much that it can actually bring quite a lot of arguments that can, you know, be in a way of you communicating and working effectively with others. So, as I started working for a global company, I very quickly realized that we all speak English and kind of with the language comes the culture of the Anglo-clusters that everybody working with English assumes needs to be, you know, the culture, the habits of everybody working in any industry using English as a primary language. But it's not really that simple. Because most of the misunderstanding that comes in day-to-day communication is not based on the fact that somebody speaks better English than another person, you know. It's mostly about differences in the cultural programming of people that have to communicate and collaborate to get things done.  You had to be smart and have like high IQ to be successful in any way. Then at some point, emotional intelligence became a thing and you had to be good with people.   And this is what we're going to talk about today. There's a Danish social psychologist that I really like, but I can hardly go through like a full lecture of his. But I like this definition that he does, that culture is the software of the mind and it's the operating system that invisibly runs your life without you even realizing it. It's what your habits, your behavior come from. It's the way you've been programmed to see the world. And before I dig in further, I want to talk a little bit about stereotypes because at one point or another during this talk, you're going to get a little pissed and be like, well, if you just know the people you're working with, if you know who they are, if you know what they're all about, if you get to know them better, you're not going to have to deal with this and you're not going to have to put them in boxes and judge them by the environment that built them. That is true. And these arguments are really valid. Speaking of cultural differences leads us to stereotype and therefore puts individuals in boxes, should be looking and judging people as individuals. That's true. But ignoring cultural context means always judging everybody with your own culture as a benchmark. And that is the big difference. So cultural intelligence is actually an outsider's seemingly natural ability to perceive and to kind of understand the behaviors of people that are not like them as if they are like them or as if they're compatriots. Over the years, I've been focusing on two main roles. I was helped before with all that was read before I jumped on stage, but I spent almost eight years of my life now at Human Made, which was a small work-press agency ten years ago. Now it's a big work-press agency. And I also was involved for a very long time with the WordPress Polyglot team.  Over the years, it grew into this global, cross-cultural, multi-regional, work-press enterprise business. And I've been really privileged to have been a part of that growth for the majority and gone through like the processes of understanding how we can work better even using our differences as an advantage. But before that I was Human Made. I was a work-press and the work-press polyglots team. And those are the people that taught me the most. Even though I couldn't put a structure around it, I put a lot of experience around it. And, yeah, luckily later on in life I kind of was able to also use that experience in my work at Human Made. And when I started with the WordPress polyglots, I didn't know what I was doing. And all I can think about is just like, how can you know all things about all people, you know? When Ze, who was leading the polyglots team before I took over, stepped down, I was just like, how are you doing this? I don't know nothing. How am I going to do this? He said, Petja, people just want somebody to listen to them. I said, oh, thanks. Yeah, that's really great advice. I still know nothing. Turns out years later, getting into something like this knowing nothing is quite a bit more useful than if you had gotten into thinking that you know everything. Because you can't. You can't know all things about all people, but you also don't have to. Really don't. Because in the base of developing your cultural intelligence, it's the ability for you to suspend judging everybody else's behavior based on what you know and what your habits are and what you've learned, counting to 10, and then maybe thinking before you react.  Why are we like this? All right. Did I already speak about the global cultural clusters? If I haven't, I'm going to forbid. The global cultural clusters that David Livermore speaks about are over there. I'm not going to focus on geographic location because culture is really not all about geography. In fact, I was chatting to Catam who is where you come. Over there, the other day, he was helping me with some stuff about this talk and he shared that he feels way better embedded and understood by his work colleagues, like his work friends, than he does by most Americans from the US. And that's normal because there are cultures and levels that are different than just geographic location. We as wordpressers have a shared culture and it's very usual for us to understand each other quite a lot better than we would with anybody from any other industry just based on that. So it's important to also acknowledge that these are very high level and it's absolutely okay for you to be born in one place but care a different culture because of where your parents came from or how you were raised or what you chose for yourself. So it's absolutely okay for you to be a total mess of several cultures like me, born in that weird era at the end of the Soviet Union and then living in a country that was trying to figure itself out for years, copying some Western standards when it came to work, trying very hard not to copy too much so that you can still keep some sort of identity. And over the years working with the polyglots, working with the humans, you learn a lot of lessons, you're in a lot of confusing situations, you go through weird things that you don't understand, you try and understand them, you learn, you grow. But I've never really been able to put any of that in any type of structure before I read this book called The Culture Map by Aaron Mere who put the world on a scale, on a map with eight scales that are the main things that are different for different cultures based on geographic location but not only. These are the eight scales, communicating low versus high context cultures, evaluating direct versus indirect negative feedback, persuading principles versus applications first, leading egalitarian versus hierarchical structures, deciding consensual versus top-down, trusting, fast or experienced based, disagreeing, embracing conflict or avoiding confrontation, scheduling linear time versus flexible time. We're not going to have time to go in depth. I highly recommend if you're interested in culture to just like take a look at this book. But I want to introduce you to some of my colleagues that helped me actually develop this talk and some others that are going to be mentioned in this talk. I already mentioned Jenny. She's over there. She's from the United States but lives in France. Catam, who is in the top right corner, who is American. Jenny Wong on top, who is first-generation British citizen of Chinese descent and she mixes cultures wonderfully. Down there, that's Miguel from Brazil, one of my favorite new colleagues. I'm from Bulgaria, Vanita from India and Lorna from Singapore. As I said, we're the first team and I kind of reached out to them while I was preparing this talk to chat a little bit about their experiences across different levels of the scale. So we're going to start exploring a little bit all the eight scales of the culture map, communicating low context versus high context cultures. Can you read between the lines? Do you prefer to be taught directly what you have to do? Do you prefer a more sophisticated way of communicating?  That's the big difference between low context and high context cultures. In high context cultures, good communication is sophisticated and layered. Messages are both set out loud but also read between the lines. Messages are also often implied and assumed body language is absolutely crucial. In low context cultures, good communication is straightforward and direct. Good communication is welcome. Anything is actually welcome as long as it clarifies things. So it gets very, very interesting how low context versus high context cultures clash in a remote work environment because we can't really read body language. And for a lot of people from high context cultures, that's a very, very big part of what you're saying. How you say something matters way more than what you're actually saying. I mean, I'd say that that matters quite a lot for everybody, but still. If you deliver a very straightforward, clear message in a particular way, the way you deliver it will be taken into consideration way more than what you're actually saying in high context cultures. Whereas somebody from a low context culture will take away what you said and don't really bother that much about any of your delivery. And at the same time, you know, repeating over and over, and that's happened to me recently, somebody on my exact team, like I was trying to get a message through and I wasn't succeeding and I kept repeating the same thing. But this is what my outcome is and I got to repeat the same thing over and over. I heard you the first time. I'm not a little child, Petia. And I was just, okay, it's not why I was doing it. I was trying to say very straightforward what I want to do. Do you agree with that? It's difficult in those situations. It's also difficult working with people from our Asia Pacific region or with clients from our Asia Pacific region who never show their faces on camera unless they're dressed for work. Which during the pandemic, during the pandemic was not a very, very usual, you know, situation. So you end up working with avatars. There is a very straightforward, according to all the books way of managing across multicultural teams. And that is defaulting to low context. Straightforward messaging, etc. It makes sense. It really makes sense. But what it doesn't make sense of is that not everybody always speaks up. Not everybody will always say the things that they think about. Not everybody will always be vocal. And it very, very much depends on where they come from, around how they express themselves, even in writing. And some people might be limited by this form of communication. So it's very important that people that don't come from low context cultures are invited to those conversations. I remember this is actually a story for another slide. Let's not get carried away with this. Let's look into this one, which is my favorite one and the one that confused me the most because I had to face it head on. Human Made is a very egalitarian company. It would never occur to anybody to call Tom our CEO, anything else but Tom. Some people don't really know his last name and get confused with other Tom's in the company. He just strolled barefoot during company retreats and in his swim trunks during State of the Humans, which is our version of State of the World. I've seen him wear the same pants for eight years in a row. He's a very, very, very low-key guy. And it was like obviously the founders kind of build the culture and the motto. It's very normal for us to be informal in our communication at work to kind of speak like that. And usually that projects quite a bit towards our clients. We would call our clients by name, we would speak to them like at first name basis, like we would call our stakeholders and even like in written communication, we wouldn't really use our names that often. But that could not be the case when we started working with Japan. When not only we were meant to be dressed for work, but you know, there was a very formal way of having to communicate with them. And we had so many clashes in the beginning of those relationships, so many things that we couldn't understand. And because our American and our Japanese and our Asia-Pacific teams had to collaborate. Can you hear me? Yeah, okay, I thought I lost myself for a moment. There were quite a lot of confusing moments that we had to figure out. So the biggest thing for me with the low versus high power distance cultures was when I had to hire my team. Because I did allocation management that the human made at this time, I needed a team that could cross over the world. I needed to hire an APAC and I needed to hire an American because I'm in Europe. We can cover the regions. And I hired Shannon, who lives close to Chicago, who's lovely, and Vanita, who lives in Bangalore in India, who's also lovely. And they had so much in common in their enthusiasm and passion for the world. They really love people, they're very outgoing, you know, so much in common. And they had such fundamental differences that it hit me really hard how much of a different approach I had to have with both of them so we can actually work effectively. Shannon and I could deal with it, you know. Five years at Human Made had taught me how to deal with Angla cultures. You know, people are communicating directly, you know what to expect, you know how to talk to them, all that. But working with Vanita caught me off guard. Four days in, we were in a meeting and I'm giving her feedback in front of the whole team about something that she had prepared for the meeting. I was like, you should do this, you know, in this way or this way, like this is probably better done in X, Y, and Z ways. And it's very, you know, it's what I consider like very mild, kind of not critical, but just like a small correction an opportunity for her to learn to grow for the team to know what we're talking about so, you know, they can help along and know where she struggles. She grows quiet 30 seconds after I started speaking and doesn't speak until the end of the call. She only says okay and keeps nodding with this expression on her face. We go into a one-on-one because I sense that something's different. We go into a one-on-one immediately after the call and she starts crying.  I was like, what? What are you talking about?  Like, you just told me that I didn't do something right. I was just like, but this was just, you know, we talked about this. We talked about giving feedback in context and you know, we're just like working through issues as they were happening and like this is the way. And you agreed. You said yes, like I like to work like that. And this is like, and of course I will agree. You're my boss, like you're telling me how things are done. Like of course I will agree.  Like you can't really give people feedback in public unless you intend to do such and such action in India. And then I got it. India is one of the most high-power distance countries out there. It was normal for her to think about critical feedback in public, meant that she was in real trouble. And she still has to this day difficulty giving me feedback in public. They do not speak to anyone that they report to in public or do critical feedback in public, regardless of how transparent the company culture is. She has difficulty giving me feedback at all. Had to drag it out of her using ways that might not be straightforward. And she still to this day, not to this day. So we've worked through these issues, but like at least in the first six months, like she really wanted me to run everything that she was doing through me. She needed permission to do any part of her job. And that was inexplicable to me. Autonomy is like how human-made became successful. We have it built in our culture and all that. But autonomy has to be taught to some people that come from different cultures. You have to teach them not to ask for permission. The head or the heart, how do you build trust? Apparently, you can't do business in Japan without going to karaoke. Did you know that? You can't. And you can't do business in Portugal without overeating. I also didn't know that. And that is true for Italy and any of the European or Latin American countries. Well, I didn't know that. So when I went to Japan, because of the jet lag and everything, I tend to generally kind of skip social events at work camps. No, I said no, no. No, no.  Like, you get that mic and just like scream into it. Do whatever. It doesn't really matter. What matters is you have to be there and then everything in the morning will be fine.  But yeah, I don't know. You can at least maybe pretend. I don't know. But the thing is in the West and in the Anglo-clusters, trust is built based on tasks. You will do what you said you will do. You're very professional at work. You're always on time. You communicate clearly. You're organized. You're fine. I trust you. Where in the other cultures, it's not really like that. You've got to go do karaoke. And multicultural remote teams need FaceTime, because that mix of people is really built. The trust can't really be built. You can't really see each other. Decisions are made by those who show up. I heard that it was an American core contributor giving a talk at one work camp once upon a time. And it turned out it's a sorkin quote from the West Wing. Perfect. And it was like a principle of WordPress. Really great. But then over time, you observe chats on Slack, on the WordPress Slack, and you actually understand the decisions are made by those who show up. And they're to speak up and actually participate. And the two types of decision-making across cultures are consensual and top-down. And if you have a multicultural team, the first thing you have to decide is how you're going to be making decisions. Otherwise, you will go asking the room. Some of the room will speak. Other people will never raise their hands. And the worst thing that you can do is in an established consensual decision-making environment to go in and force a top-down decision. Trust. If you're in Japan, you can go to karaoke, but I don't know what you do if you're anywhere else. We're trying, like, embedding the concepts of high-performing teams of human-made right now. And one of them, one of the principles, is that you have to embrace conflict as a productive way to move past an obstacle. All men was our APAC team freaked. So many people in these cultures don't really do public conflict. They don't. And in talking to Lorna and Vanita about it, like, there are principles deeply embedded in the culture that say that, you know, you resolve things privately. You resolve things between individuals. You can't, like, have a public fight in a room and then be okay.  It's not an issue. Like, he doesn't really consider that to be problematic. It's the most usual thing to sit down with, you know, the Portuguese and the German to have, like, a really heated discussion. I mean, the Germans don't really get heated. All you get is this. Well, they argue, but still, you know, they get heated in their own way. And you disagree and you, like, scream. I mean, some scream, some, like, very, very, kind of earnestly disagree.  It's fine. And some cultures, if you do that in public, there's no way. There's no way you want to see this person later. There's no way you want to work with them, any of that. So embracing conflict is quite tricky. So you think about that. Just don't imply everything. Don't apply everything that you want to do in a cross-cultural company in the same way throughout your regions. Speaking frankly, is that a gift or a slap in the face when you give feedback? Is that a direct negative feedback? Do you sugarcoat it or not? This deeply matters. And the why versus how? Do you start with how you're going to solve an issue or the theory around how an issue should be solved? Those are the other two. I'm running a little bit behind, so I'm not going to get into detail on these ones because I want to talk about this one. Time. So in linear time, project steps come one after another. Deadlines matter. Everything is fixed. And we know what the cultures are. That's really, really what the cultural clusters are that really, really depend on that. In a flexible time, flexibility is everything. Project steps are approaching the fluid matter. The focus is on adaptability and not on everything being done in a particular order. So, sorry. So on the scheduling scale, we've got Germany and Switzerland on one side. We've got India and Kenya and some of the Asia clusters on the other side. And I, and somewhere, I guess, around the Russians on like a little bit of the fluid side, but don't really know where exactly. Because the culture map is a scale. It really depends not only where you come from, but like who you are. What are the cultures that influenced your growth? And that's something that you shouldn't be forgetting. So how do we do this? How do we know all things about all people? How do we develop our cultural intelligence? It starts with figuring out what your core is. What are the things that make you you? But maybe, really, you. What are the things without you're not going to be you anymore? And what is the flex? Where is the flexibility that you have to adapt and to grow? On the left hand there is the salesman that will flex to sell you anything. They will not have absolutely any core values. And you won't trust them for the world. They don't have anything that makes them them. They will flex towards you. On the other hand, over there is granddad who will not flex about anything. He is who he is. This is the way he does things. And there's no moving that. There's no moving the scale. And in the middle there is your ability to adapt towards the otherness of others. The big thing here are the knots. Because as you start looking into yourself and developing, you actually find things in your own culture that you're not particularly proud of or comfortable with. And those are your knots. And those are the things that you probably have to work on the most in order to be able to collaborate and understand others and understanding whether you can actually adapt them and move on from them or they're just an integral part of you. And you just have them in mind. You keep them. You keep them. Some of the things that I figured out about myself like a while back were the things I'm uncomfortable about is like I'm not really comfortable with where I'm in life, not because I'm not happy at where I'm in life, but because I've been culturally programmed to be doing an exactly different, like a really different thing at this stage. You know, babies, family, what my purpose in life is shouldn't be this. I'm really uncomfortable with that, even though rationally, rationally, I don't have an issue with where I am. I am happy. But inside, it struggles. So it's work every day. And cultural intelligence is developed by having the courage to actually ask the uncomfortable questions and dig into those things that you feel deeply uncomfortable about. Have the willingness to admit what your knots are and either move past them or acknowledge them. The paradox is you can only develop cultural intelligence if people will share with you and they will only share with you if they think you have enough cultural intelligence to understand and be willing to listen. So be willing to listen. Turns out, Zay's advice on point, people just want somebody to listen to them. Petja, you said, be genuinely interested in other human beings and their experiences and the otherness doesn't have to be otherness. It can be what's interesting about a person. Be aware of your knots. Learn to make a fool of yourself. It's useful in general from someone very particular and tied up. Recognize and set up the cultural intelligence as well when you recognize it because staying silent is not going to help as well. Somebody recently told me most people, Petja, listen to reply. I try to listen to understand and it's stuck with me and I think that that's the core. When you listen, listen to try to understand. You might not understand, but at least you will learn because a lot of people think about cultural intelligence as knowing a lot about others and what it actually is is knowing your own culture starting to decipher that. Reverse engineering yourself a little bit so that you can react in certain situations. Spend the 10 seconds, not carry any frustration or anger or misunderstanding or confusion with you more than you have to because not everything is resolved in 10 seconds. Let's be honest. So how over time am I? 44 minutes almost. So what did I do about being on time? Because I quickly realized that I was pissing off more people than just Jen. So I set up my notifications to ping me a minute before a call and I would go on the call immediately be the first one in regardless of what I was doing while I was waiting for everybody to get in because the thing was that I quickly realized being on time is in my flex. I don't care about that. I don't really mind. It's my cute little work from when I was a kid. It's something that I think I can get away with but it's in other people's core and that's important. And it's a little thing that I can do to change. And I'll be honest. This is a very, very easy and straightforward kind of flex example to be giving. There are things that are going to take more than just changing a notification setting for you to kind of get over. But it works. Because being on time is in a lot of my colleagues' core. And I may be late every now and again but not as often as I was. And now, before I get over time because I'm in the US I'm going to say thank you. The slides from this talk are on my speaker deck and on the Human Made blog there is a summary of this talk that has some resources from the material that I mentioned on the talk if you're interested in learning more or reading more. So yeah, do we have time for questions at all? Two minutes? Okay. Yeah, I think there's a question. Thank you so much. It was a really great issue. Thank you. My question was a little bit about I know that there's a lot of talk about toxic work cultures in general. So we do have a lot of responsibility as leaders to ensure that our culture in the office is good for everyone. But you also mentioned that there are certain things for example deciding that you might want to set up as your company's culture to begin with before you start onboarding other people and adjusting for their cultural specifics. So is that the same for many things? Or is it different in your opinion for the different areas? I think it's definitely different and it helps to have somebody very culturally aware in the company to be able to kind of go through and implement these things even on regional levels. And it's okay for a company culture to start from one spot and then evolve. What's important is to not try and box people into a culture that was developed from the one global cluster. It's still fine to set up ground rules but it will take you longer to onboard certain people towards that and you have to have the patience and give them the time to adapt and not expect the same results from people from different cultures necessarily when they're culturally sensitive issues because it's like asking a person who can swim and who cannot swim to swim at the same time when they're asked or they will drown where the other might not really have a big issue or a big difficulty with that. So yeah, it's worth looking at the eight scales, figuring out what are the core values of a company and the same thing as looking into yourself. What is your company's culture and what are the core values? Which cultural clusters are they best tuned into? Do some of them need more attention and where are the parts of your onboarding process that need tweaking based on where the people are? Does this answer your question? Okay, anybody else? There's a question over there. Thanks for a great talk. I myself a little bit here. I worked in software global in the mid... Sorry, the mic is kind of cutting off so I'm not sure. I started working in globalization of software in the mid-90s and working with teams and managing people to make their software ready for global markets. And what you presented today was actually the bigger problem in terms of getting software ready for global markets is the different cultural aspects. And this is a great presentation of kind of capturing that problem. The software side in some ways was the easy part and getting people on board to make sure that the software developed and worked in Japan versus America was technically fine but the subtleties of all the cultural differences. And I've been in this area for some large number of years now and I still find it frustrating that we're still having to have people like you give these talks because we haven't yet as a community really fully embraced what you said. And I'm wondering if working with your clients, whether or not you're seeing improvement in this area. Definitely. We are very... Well, the thing is when you're doing business, the shift and the adaptability of like how your employees kind of see things adapt as the client needs to adapt. So we adapt to our clients better, towards our clients better than we adapt towards our people. This is one of the big things that we have to kind of change I think as an industry, because like we are very ready to accept our client's cultural differences and adapt towards them. Maybe not necessarily ready all the time, but like at least we're very quickly become aware of them and try and adapt, and we're way slower adapting towards like the people that are actually working for our companies. So yes, I see a lot of adaptability towards business relationships. I think it would be great to see more adaptability towards like the people working in companies as well, because that serves both. Does this answer your question? Thank you. All right. There are two more questions over there. Do we have time for them or if not, I can also absolutely, I'd love to kind of have conversations in the hallways. So please find me. You know, I will be around all day today and not stressed anymore. Thank you.


Holger Ruiner vs. Carlos Alcaraz is the second quarterfinal for the Paris Masters for 2022 and both players have had different roads to get to the quarterfinal stage. Ruiner started the tournament off unseated and will take on Stan Vavrinck at the veteran in the first round and after losing the first set he had to come back and fight for the win eventually beating Stan in three, four, six, seven, five, seven, six. In the second round Ruiner would take on the number 10 seed, Hercatch, who's playing for an A2B finals spot but Ruiner would take out Hercatch and take out his chances of making the A2B finals, winning in straight sets, seven, five, six, one. In the third round he take on another player who was playing for an A2B final spot, number seven seed, Rublev and again Ruiner was not intimidated by a top 10 guy beating Rublev in straight sets, six, four, seven, five to advance to the quarterfinals. Alcaraz came into the tournament as the number one seed and for the first time in a big tournament playing as the number one player he forgot to buy in the first round. In second round he took on Nishioka but this was a straightforward match up for him as Alcaraz prevailed in straights, six, four, six, four. In the third round he take on the veteran Demetrov and again just like the Nishioka match Alcaraz had no trouble getting through in straight sets, six, one, six, three to get to the quarterfinals of Paris. These guys have only played the one time before, that was at the next gen finals at the end of last year with a weird sort of scoring system and Alcaraz won that match but I don't think there's much we can look into that one because of the way they score that being at four, two, four, three. It's not really a proper match so they really haven't played a serious match yet. If Runa is going to win he has to be fearless. Both these guys have that fearlessness in them but Holgaruna doesn't have anything to lose. Alcaraz is the number one so he's expected to win. Runa's got to come out, be super aggressive, they know each other's game well because they are friends but Holgaruna he's going to have to be aggressive and not be on the back end of that Alcaraz aggressive game. If Alcaraz is going to win he needs to be the one who gets aggressive first and also we need to see how his knee holds up because he has been having trouble with a knee this week which could be a weakness so he has to be more aggressive than probably usual because he doesn't want to run around too much against this kid. This is going to be a very fun matchup. As I mentioned they're both friends they're both kind of growing up together both 19 year olds very exciting players but I'm going to go with Alcaraz. I think experience is on Alcaraz's side just a little bit because Runa hasn't really made it to the heights of heights yet he's had a very good year but Alcaraz had a better year and I'm going to go with Alcaraz in three sets but it's going to be fun fireworks two fearless players going at it who do you think is going to win this matchup?


It's probably not surprising for you that during the time the amount of data grows almost exponentially and with the internet of things and more and more devices connected to the internet we can be pretty sure that this trend will continue and probably in a couple of years the amount of data being generated on the internet will again grow exponentially. This is probably not surprising. What is maybe a little bit more interesting is the structure of the data. The black part which grows linearly is a rough estimation that this part is formed by structured data but this exponential part is mostly created by semi-structured or unstructured data. This has some implication and I will return to this a little bit later. I guess this amount of data is really big and this phenomenon is known as a big data. For everyone it can mean something different. Maybe to be sure that we are on the same page how big the data has to be to call it big data. Do you have some definitions or how big the data has to be to call it big data? Some more tips. I have scarf, I have half full bottle of kofola if you don't own scarves or banana. Terabytes. Okay, terabytes. Something somebody else? No. One megawatt. Okay. So some definitions. Okay, so probably the most popular and commonly used definition is this one, that big data is anything which carries your Excel. It's kind of joke but we can take something from this and that is that basically you can buy more beefy machine, you can buy a recent version of Excel which supports more throws with this exponential growth. Sooner or later you will probably, the scaling up won't be sufficient and you'll probably have to scale out. And this scale out and scalability issue is not always just because of amount of data but also because of cost efficiency. For example imagine you run a shop on the internet and before the Christmas you probably have to deal with really huge amount of data while during the summer when everybody is on the holidays the amount of orders will be probably significant slower. So there's no point to buy some really beefy big machine just before two weeks before the Christmas and the rest of the year have this beefy machine. So there are two main reasons. Just amount of data and not only this but cost effectiveness. And some other challenges of big data. This is not completely a couple of challenges which I will try to address later in this talk. So if you have huge amount of data, of course running an analysis on top of that is challenging just because of a huge amount of data. You have to also store somewhere the data. And as I mentioned in the first slide quite a lot of data can be unstructured. So you have to have some solution which is able to store the unstructured data or data which are not processed. Again there can be other reasons like that the data have some structure which are not able to process them and just for performance reason you store it as they come and process them later. And when you run some analysis on top of that you probably will get some data which has some better structure and which you will probably want to later on run some queries on top of them. So you probably also need some solution which is able to store some structure data and run some queries on top of that. Quite often you need both store unstructured data and as well as structure data. The solution as I mentioned has to be scalable and the scalability is usually done in the cloud. But running application in the cloud gives some more challenges because it completely changes the architecture of the application. In the cloud everything is ephemeral.  So can't rely that I will store the data here and two days later the data will be there. So it changes also the approaches how we create application. So probably the most widely used approach is how to address this kind of challenges is data replication and then running some MapReduce on top of the data. And probably the most popular solution is Hadoop. So far so good. I would say I just recap some probably well-known facts which you probably know. So are there any Hadoop users or who run or at least play with that? So are you happy with Hadoop or something? So it seems everything is okay. So why Hadoop is not good enough? So what doesn't make you that you are not completely satisfied with Hadoop? Well, you know people today are more and more impatient and want their answers immediately. You know nobody sends letters, everybody sends email or some people even make phone calls and want to have some communication in real time. So for some people it's slow because you store some data in Hadoop FS and for example once a day some around some analysis on top of that and some people may and with the analysis take pretty long time. So some people seems wants to do it faster. So how we can speed the whole process up? So one little powerful idea is especially for algorithms which are iterative don't store the data to Hadoop FS or to some permanent storage during the computation keep the data in memory all the time. And as I said you remember that you run it in a cloud. So it has some implication you are replicating the data you run some transformation on top of the data and if you store it it will also get replicated and again it takes time. So maybe stop and think a little bit do I have to replicate every single change and the answer would be probably no.  So it's easy to say maybe a little bit more hard to do but fortunately there are some frameworks which already does that and probably the most popular one is Apache Spark. And a half of Apache Spark is concept which is called Resilent Distributed Data Sets and the shortcut is RDD. It's basically immutable distributed collection of data. Immutable it means that if you run some change on top of that it doesn't change this RDD but it creates new RDD and there are basically two kinds of operation you can run on top of the RDD. It's transformation and the transformation takes one or more RDDs and create new RDDs from it. Typical example is map operation or filter operation. Second kind of operation is action and actions it takes some action as the name suggests on the RDDs. For example count how many items are in given RDD or take the first element and so on. And now the important thing is that RDDs get evaluated lazily so that you applying change of transformation and it does nothing. It runs in millisecond because it just locked operation you want to do and the RDDs gets evaluated only when you call action on top of that. So there is really huge space for optimization. Imagine that you run for example five transformations or last one is filtering and the action is called the action is calling first element that you want to run some transformation and are interested only first element. So in let's say classical approach you would run first transformation of whole bunch of data save it somewhere second, third and then you pick just first element. So basically obviously if you know what you want to do there is pretty big space for optimization so that you run the transformation and why once you filtered first element which matches the last filter you just return it and can stop and that's not all. Also as for replication you don't have to replicate whole data you just replicate this sequence of actions it's called lineage and when some partition of the data is lost what you do you take original data or some RDD which was stored in previous time and recompute the data. So basically you don't replicate every single change and the only thing you replicate is lineage and when some crash happens the data are recomputed and it's again safe. It shows that this saves quite a lot of time and not only that the RDDs are kept in the memory if possible so it brings really huge speedup. This is the original paper about RDD it's quite interesting read so I recommend it to take a look on it. So this is plot from Spark web page and it's comparison to run some machine learning algorithm for classification on top of the Hadoop and top of the Spark and as you can see the speedup is two orders of magnitudes it's really huge speedup. I'm not saying that for every application you have the speedup will be two orders of magnitudes papers report that usually the speedup is one order of magnitude and for iterative algorithms it can be two orders of magnitudes but again it's really nice speedup so I think this is quite impressive and really worth to think if your application can be pushed to the Spark and you can take benefit from it. So it sounds good but can we do even better? I believe at least for some application yes. Let's try to do the processing of the data once it arrives. Maybe it's quite trivial a suggestion but if I ask how many applications really do that I would believe that not so much or not so many and maybe it sounds trivial but the implementation could be a little bit tricky but again fortunately somebody already did that and for example Spark provide streaming support for streaming and as the data comes it creates from incoming data RDDs and put RDDs into micro batches and it emits these micro batches to Spark workers which do the processing. User can here configure how frequently the bunch of these micro batches should be created. It has some advantages but also some advantages to do this micro batching but usually works pretty well. So if you have some again it's immutable so from this is for example when incoming you have incoming stream or sorry in Spark terminology it's called the discretized stream or D streams. So here you have incoming stream of lines of the text and some micro batches and imagine you have you run some this transformation on top of that that you split every line into single words and create a flat map from it. So what it does it again creates new D stream and from applies this transformation to every single batch and so you will get new stream of this micro batches which you can process. So if you rewrite your application to Spark switch to Spark streaming is pretty trivial you just do some couple of changing lines changing context to streaming can switch from RDDs to D streams but internally these D streams again contains RDDs so there's not so much work to switch to streaming and do the processing of the data in real time as the data comes. If you are not happy with my... Okay you probably have to scale your solution or add more brokers. Yeah but it's the advantage of running your solution in the cloud so for example if you run it on some Amazon you just pay for what you really use so during the day you will scale and bring or create really huge cluster and during the night you can slim it down to one node so basically this is really advantage to run it in the cluster so basically application have to be prepared for it so we have to be scalable and Spark definitely is there can be some other challenges like that if you hard code something here so it can happen that this wouldn't be enough and you would like to switch for example to one millisecond so this is one of the disadvantage so here's my answer maybe you can switch to some real time streaming processing frameworks probably 3 well most well known is Apache Storm, Flink and Samza all of 3 are pretty similar originally I wanted to give some introduction to Apache Storm but I found that I don't have enough time for it so fortunately you won't lose the information because it's homework for you and once you go home first thing you should do is to open the web page and read and learn some hello world example of one of 3 all of 3 of them are pretty similar so it's fine to read for example some hello world for Apache Storm and to have some feeling how it works it's not something super complicated so it shouldn't be big deal to understand how it works but it's pretty powerful okay and can we do even better so at least in some cases I believe yes and what can we do more okay Spark keeps the data in memory during the computation but you usually don't store the data run some Spark analysis on top of that and then finish and throw the data away usually you have some whole stack of applications incoming data for example some Spark processing and the result of this analysis is just recent to another application to which does something else for example it can be some other processing of some already prepared data and this application send it further to another application which can be for example some business processing engine which can take some automated actions based on this data and so on so there is usually whole chain of application and you have to exchange the data somehow between this application usually store it somewhere in Cassandra or some SQL database and so on but let's try to keep it in memory all the time so how can you keep the data in memory all the time during processing whole application stack the answer is to some caching is that caching is some sketch how cloud works and you see that caching is a basic principle how cloud works because it's basically just caching but to be more serious the answer is to some in-memory data grid like for example in Finishpan it's pretty major in-memory data grid solution I won't come into the details because first I don't have time for it and secondly there was a dedicated presentation about Finishpan this morning so if you miss it I really recommend you to take a look on the presentation on YouTube or if you just want to have some quick review go to Finishpan.org and check some features I'll just quickly mention that it's no SQL key value data store but you can also define some scheme if you define some scheme you can run indexing and queries on top of the Finishpan so this to some extent address the challenge which I mentioned at the beginning of the presentation that you sometimes need some solution which is able to store no SQL data but at the same time able to store some structured data and run queries on top of that Finishpan could be a pretty good solution for this of course it's scalable, elastic and so on there is no single point of failure so if some node dies it data are replicated and everything works well one nice thing is it supports transactions so if you run some for example financial application and really needs to be sure that data arrive where it should it also supports transactions and have quite a lot of nice features I will talk about some of them a little bit later but most of them please go to Finishpan.org or check the presentation of my colleague Kierka Haluša from this morning so how we can connect it to Spark and things which I talked before there's a connector for Spark which connects Spark and Finishpan what does it mean? it means that you can pretty simple just by two lines of convicts read and write the data from and to Finishpan from Spark cluster so here is just that you define some Finishpan server address and name of the cache which you would like to process and simply create RDDs or stream from Finishpan data and maybe you can have some different use cases for example data are coming from Kafka are processed from Spark and you just want to push the data for processing to Finishpan so you can of course write the data one nice thing I like to mention is that you can transform RDDs by using Finishpan queries so imagine for example that you store some users into the cache and now you are interesting only users which has the same name as I have so you call the query on users having name equals to Wojtek and then you filter RDDs just by running Finishpan queries on top of this R of RDDs and everything happens inside the Spark so it's pretty nice pardon? everything is kept in memory pardon? the question is if everything is kept in memory so the answer is yes actually in Spark you can define some storage levels so actually in Spark you can keep it also on disk but what is highly recommended is use storage level memory only maybe you can run into some troubles if you use this option because it can Spark can run out of the memory but it depends on you so you can store it also today on disk basically if I remember correctly or you can use another strategy memory and disk so Spark automatically tries to keep the data in memory and if it's not able to keep it in memory it starts storing the data to disk but you have two solutions you can scale up and add more Spark workers and if it's too costly for you you don't have to have the data right now so you can store it to the disk and process it later from disk but it would be a little bit slower but it will cost less money so it depends on your use cases and how you evaluate your application but basically you have both options you don't have to do any compromises no but it's recommended solution recommended solution is to run on every or you have machine where you run Spark worker and recommended approach is to run on the same machine InfiniSpan or one node of the InfiniSpan cluster because the connector is clever enough and it loads only the data which are stored in this InfiniSpan cache segment which runs on this machine so basically yeah it's not just that InfiniSpan and Spark shares the same memory it's load over the wire but over the local host so it should be pretty fast so it's recommended solution to run it on the same node it's not mandatory but of course if you would run it on the different machine you will have latency on the network but recommended solution is to run it on the same machine actually both because yeah no yeah for processing yes because the question was if the data lives only in InfiniSpan or if I double the data if I load it into Spark are functionally yes because these are two different processes which doesn't see the memory of second process so you have to load and so yes but there's some assumption that not of whole data will be load into the Spark so let's say there is assumption that InfiniSpan will play as more permanent storage and you will only load some let's say stream part of the micro batches into Spark processes so away and so on so you don't have to replicate whole data which are kept in InfiniSpan okay so couple of other interesting InfiniSpan features which can be used for pushing the data so imagine that Spark processing is done the data arrived and you like to push it to other application down the stack so there is pretty easy solution for that InfiniSpan has client listeners so whenever anything change in InfiniSpan cache it sends you the notification that something has changed so you can take immediate action based on it some little bit more advanced concept built on top of that is continuous query imagine that your user runs some query on the data you get from your analysis and once as the data are coming he wants updates so you have to open for example web socket the query again and again but with continuous query just registered the continuous query and when some data arrive which matches the query you are notified about the change and can immediately show the change to the end user or some other application and whatever and another nice feature is implementation of distributed streams is basically implementation of just streams but over the distributed data so if you have some application which only do for example some map introduce you can avoid use spark and run it directly on InfiniSpan so in this case you would completely avoid doubling the data but of course spark provide much more functionality because it's dedicated tool for it for some more easy or simple use cases this would be to use only the spark so what do we do get from the big data we try to keep the data in memory all the time which should speed up processing of the data and exchanging data between the applications you process the data as they arrive and result are pushed to the users by immediately as we have it by for example continuous query so can we call it fast data well I don't know because DevOps borups stopped tweeting several years ago so I have no definition for fast data but actually I think it doesn't matter how you call it if it makes you happy to call it fast data let's call it fast data but what really matters is if it makes your user happy and if your users are happy you will be probably happy too and I believe that applying this techniques will make your users more happy and this is what matters I'm not saying that it's a silver bullet which fits to all use cases so it's just a couple of sorts how you can do better and now it's up to you to think about your application and to do the decision if you can somehow benefit from it if yes, the tools are ready so just try to use them so I have only not so much time so I have some demo it's really a hell of an example really trivial I will probably run through the code very quickly because I don't have much time but you can download the presentation and go to my github and the code is there so you can go through the code it's well commented and it's really few lines of code so you shouldn't have the problem to understand it it basically tries to show you or imagine that you have some network of sensors which measure temperature send it somewhere to some gateway which stores the measurement into infinite span then you want to compute every temperature for every place and you would like to push the result to the user so basically it does nothing useful but it's again a hello world example so as every hello world example in the cloud it has unfortunately several components so the first one is the infinite span server it will store all the data incoming and outgoing then there is one application which simulates this network of sensors it randomly generates some capital city in Europe and randomly generates the temperature and send it to the infinite span then there is spark for processing the data and computing temperature average for every place and then there is some client application if I want to be really in a cloud I should have infinite span cluster and spark cluster but let's skip it for now oh sorry so I'm starting infinite span server my spark server is already running so where do I have it it's here so you can see I have one master and one worker on my localhost every synchronous on my localhost so so infinite span is running so now I will start generating the temperatures as you can see it generates some random city and random temperature I will I will also start my client here I'm passing some arguments to it that I'm interesting in temperature only in Prague and Vienna for example so I don't have all the changes so now it waits if it gets something and here I will start spark streaming and it will will listen to infinite span and process the data as they arrive it prints how many data is in every bunch of the stream so it's generated around 10 items so let's take a look and here you can see that I get updates for Vienna Prague as they come so yeah it's changing so a little bit let's take a little bit look on the source code here is the client as I mentioned it's just simple cash listener for infinite span so I just the only thing I need to do is to add this annotation that cash entry data created or cash entry data modified and that's all and then I can implement my logic which does only check if the city which is updated is among watch cities and if yes it will print something to output and as for spark application it's simple it just registered to infinite span and create infinite span D stream from it then does some processing like extracting the city and the temperature then it as I mentioned in spark everything is done in microbeaches so it can happen that you will get several measurement for the same place in one microbeach so you group it by the key and then you just recompute the average with some math function and obviously as it's stream of the data so I need to keep somewhere the previous state so I will keep number of measurement done during the whole period and some of the temperatures and from it compute the average and I need to keep it somewhere spark provides nice feature for it it's called state so you can run the map with state and it's here so I'm storing for each city I'm storing this sum and what I basically do for every measurement I update the sum here and update the state and from this update I compute the average and return it to the new stream in spark with averages and later on I take this stream of averages and create it back to the infinite span which store the data and fire my listeners in another application which will print it to standard output so that's that simple again I'm sorry I'm running out of time so please check my github and you can go through the code if you want more so what to keep in mind try to think if you can keep the data in memory all the time if you can process data as they arrive if possible keep the data in application during the whole memory processing the whole application stack and infinite span and spark provides really nice features which you can use so these are some sorts you can think about and if you have any questions I prepared some with answers but if you have some other questions so we have a couple of minutes thank you I would like to ask what's the structure of the RDD when I have infinite span to store the data I have just the keys to the infinite span so if the RDD is transformed so it doesn't transform with the data but just the keys no it contains key and value so pairs of key and values for RDDs and for stream is key value and kind of type of operation so if the data was removed, updated or created so with the operation for the transformation are stored into spark actually for transformation are not stored anywhere data are processed only when you take some action off on it and then are stored in spark memory and if you want you can store it back to the infinite span thank you I would like to ask how is it guaranteed if I have multiple instances of spark and multiple infinite span that every single item is processed only once because I have replicas in infinite span so how is it guaranteed? it's by primary owner of infinite span so basically you take only from primary owner if it's not available it talks to the replicas but basically it takes only from primary owners thank you Wojtek for great presentation okay so thank you if you have other questions catch me here thank you


Tych first with 30 minutes anyway, but I still have to talk about something first that's more important than python packaging. That's me! So my name is not GG2K, but this is my handle more-or-less everywhere online. Feel free to follow me on Twitter. I tweet about python and Brexit occasionally, angrily. So, yes, my real name is Mark Smith. I'm a developer advocate for NEXM換, one of the conference sponsors. I will be remiss if I didn't at least briefly Ie, golyda i gael o Nexmo, mae'n ysgolwyr a'r ysgolwyr, mae'n datblygu ymdyn nhw i gael adegau a'u gwahanol sy'n gweithio ar y cyfnodau iawn, a'r cyfrannu gwllwni amgylcheddau, dwi'n gweithredu, i gael o'r deisydd o'r mobile a'u web apps i gael. Felly, mae'n bwysig yn ei ddweud, ymdyn nhw'n amlwg ysgolwyr o'r Nexmo oedd oedden nhw i gael o'r twid, o'n ymdyn nhw i gael o'r cyfrannu. that another time. So now I've talked about Nexmo, let me talk about JavaScript. So in March 2016, a developer removed a library called left pad from NPM, the Node.js equivalent of PyPI. It's a big web service, holds packages, when you need to pull down those packages and install them into your JavaScript application, you run NPM and it downloads them and installs them so that you can access them from your program. Left pad, so it broke lots of packages that depended on left pad. It turns out that quite a lot of packages on NPM depended on left pad. In fact it had been downloaded 2,486,696 times in the month before it was removed from the NPM repository. It was just one function and it was 11 lines of code that padded a string to a certain length by adding characters to the start of that string. And lots of people thought that this made the JavaScript community look kind of silly. It's why would anybody publish a package that only consisted of 11 lines of code? Why would anybody use a package that only consisted of 11 lines of code? Now I don't think it made the JavaScript community look silly. I think actually it made the JavaScript community look pretty awesome. Now obviously there's some problems with the fact that people can remove a package that everybody's depending on and kind of break the entire JavaScript community software but in general why was this package published? So if you don't agree that this made the JavaScript community look awesome, feel free to fight me on Twitter. So the left pad is not a problem. Left pad was a solution. So because the JavaScript standard label library is relatively small and doesn't contain lots of useful functions that say the Python standard library does contain, like the ability to left pad a string with some characters, a developer wrote a solution for it himself. And because NPM makes it easy to publish small items of code, he did. And he made that code available to other developers so that they then didn't have to reinvent the wheel. And that means that people can bug fix in one location, they can submit PRs to improve these 11 lines of code and then ultimately people can well ideally depend upon it. Now the alternative to this is copy and paste. Copy and paste I think you can agree is not how you should share your code. Because it's really easy to share with NPM people do and even with really small libraries and I don't think that's really the case with Python because people find Python packaging slightly fiddly. There are some slightly odd things which we'll go through about pushing your first Python package to PyPI. I made it difficult for myself to say some of these sentences as well. So people are a bit afraid of setup.py. There is good documentation out there, lots of it conflicts with each other, but there is I think a growing movement to bring some of this stuff together and make it easier to find current best practices. Hopefully this talk will help a little bit. So what I really want everybody in this room to feel at the end of this talk is confident to publish packages to PyPI. Relatively small packages, maybe ideally pure Python packages, which is what the example is going to be. So I would like you all to make a package. What we're going to do in this talk is I am going to show you how to make a package. So the first thing, the assumption is that you already have some code which is general enough or close to general enough that you think it would be useful for other people. So let's take some useful general purpose code like a function that prints hello world. Now who in this room hasn't written this program in some form? Exactly. Now that is totally wasted time. Everybody in this room has written this program. Wouldn't it be so much better if you could pip install it and call that code from somewhere else? So that's what we're going to do. Also note there's an F string here. F strings are awesome. You should use them. So the idea is you've written some code that you're proud of. You would like to share it with the world. So the first thing to do is extract it from your code base so that it is independent of that code base. That's your problem. I'm not going to show you how to do that. In this case we're going to extract this code into a file called hello world.py. This is a Python module. Next we're going to put that module in a source directory and I will explain towards the end of the talk why we did that. For the moment just assume that Hinnick is excitedly clapping for the second time today. Awkward. So I'll explain why later, but for the moment just take this as correct practice. In the same directory as the source directory, the container source directory, we are going to create our setup py file. You can open that up in your favourite text editor or Python IDE and we will enter something like this into the file. The first thing to note here is that we're importing from setup tools not to distutiles. You will still find documentation online that recommends importing from distutiles. Don't do that. It's not that powerful compared to setup tools. Pip already is distributed with setup tools. If you're installing packages with pip, you have setup tools. It's not a third party dependency anymore. Then we have this function underneath it. We call setup that we've imported from a setup tools module. It's a bit weird. For now just don't think of it as a function call. Think of it as configuration. Each of those parameters is essentially a line of configuration that you are giving to pip to tell it how to install your package. This is pretty much the bare minimum setup information you need to provide. We start with a name. Name is what you pip install. This is the name on PyPI that it will be uploaded under. People will pip install it. It doesn't have to be the name of the Python code that people will import. It's a separate thing. Usually they're the same. Sometimes they're different. We need to pick a version number. Here I've just started with 001. 00x version numbers imply that it's unstable. There is a good chance that the first few times you upload this to PyPI, there will be a minor packaging mistake. This is a good stage to start to upload packages to PyPI while you've still got this unstable version number. You're not worried about people seeing instability that's not actually your code base. It's actually your packaging configuration. Then we have a description. This is usually a one liner at this point. Say hello is not a very useful description. But we'll leave it at that. Then we have PyModules which is a list of the actual Python code modules. We have a file called hellowell.py. In here we're saying this is the code that we want to distribute. That's what people import, not what they pip install. Then finally, again, this is a cargo copy and paste setup config. We have this package deer line which is a map that sort of said empty string and a source. All that is doing is telling setup tools that our code is under a source directory. Don't worry. Put it in your code and forget about it after that. Now we've built a package. Let's build it. Potentially we could distribute it. We run the setup file we just created with the bidest wheel command. This tells it to create a wheel file that's something that is appropriate for uploading to PyPI. It will spit out a load of output, most of which I've deleted from here, but the line that I've highlighted in bold up here is the one that's important. What that's saying is that it's just copied our hello world Py code file into the lib directory, which means that it will end up in our wheel. If that's not there, then essentially there will be no code. It will have an empty wheel file and it won't work. We can now look at what's been created as that part of that bidest wheel command. Here are the directories and files. Remember we've only actually created two files so far. Hello world Py and setup Py. Everything else here was created by setup tools. I have a few things here. It's created an egg info directory in our source directory. You'll want to get ignore this. I'll show you how to do that at the moment. This is horrible. I wish it didn't do this.  Then we have a build directory. This is where setup tools moved our files to in the process of building our wheel file. You will see the hello world Py file in there. Again, validated that our code is actually going to be in our wheel. Finally, we've got the actual wheel file here that it's put in our disk directory. That is our final distribution. Now we can install it locally. This is just effectively testing our packaging. It's not testing our code, but it's testing our setup Py file. Here we are going to pip install and then with the minus E flag and the full stop period dot whatever you want to call it. This can be confusing to people if you haven't seen this before. Just out of interest, who hasn't seen this before? Yeah, I thought so. This is actually an essential command if you are building Python packages or rather they're essential flags. The minus E, normally when you install a Python package, it installs it into your site packages folder, inside your Python distribution. It copies the code into your Python distribution. We don't really want to do that. While we're working on our project, we wanted to just work with the code that's in our source directory. That's what minus E does. It essentially links to the code that you're working on instead of copying code into another location. Once we've installed this package, we can continue to work with it, continue to run it, continue to write code against it without having these two copies of that code that is just going to cause us problems further down the line. The full stop at the end means install the package in the current directory. It's looking at the setup Py file. It's saying install this package by linking to the code that I'm working on. You run this every so often.  Bear in mind, our code is under a source directory at the moment. If we run Python in our current directory, we can't import Hello World yet because it's not in our path. Let's, in theory, let's test it. Here, we run Python, the REPL, and then we do our from Hello World import say hello. Because we've just installed our code into our current virtual environment, this will work now. Even though our code is under the source directory, Python has been told where our code is by the setup Py file. We can execute the say hello function, we can execute it by passing at the optional parameter. Everything works as we would hope. It's some rough testing. We'll get on to better testing at a moment, but it's just a confirmation that our code is installing correctly. At this point, we have a working package with some useful code in it, so we could upload that to PyPI immediately, but I would say that there's a few things that we really need to do before we get to that point. That's documentation and testing, but also just a little bit of housekeeping which I'll run through now. As I said, there's some files created that you really don't want to add to your Git repository, so it's useful to have a Git ignore file. This website is fantastic. They make it easy to get hold of GitHub's standard Git ignore files that they publish for different language and operating system communities. You would write Python in that text box, hit create, and it will just spit out a text file into the web browser that you can copy and paste into a .gitignore file. Now we're ignoring all the main files that Python creates, so it will stop you from uploading PyC files and a bunch of other artifacts on a Python project. If we're going to publish this code, we also need a licence. If we don't have a licence, we haven't given permission to the people permission to run our code. They can look at it, but they can't copy it or use it, which is not greatly useful. We need a license.txt file. If you don't know the ins and outs of the different licences and the restrictions and freedoms they grant the software that you're publishing, this website, chooseallicence.com, is incredibly useful. It essentially asks you lots of questions and then gives you your options and how they compare to each other. It's a good way, it's a human way for non-legal people to understand the differences between different software licences. We need to add some classifiers to our setup py file so that people can find a project in PyPI by filtering on common criteria. Here we say that this is Python 3 code, it runs under Python 3.6 and 3.7. We haven't really tested that yet, but we know that it only runs under those versions of Python because there was an F-string in the code, as I pointed out. I chose the GPLv2, so we put that in there. That was a bit of an arbitrary choice. These can all be looked up in this URL at the bottom.   Then you need some documentation, but before you write some documentation you need to work out what format you're going to write your documentation in. You basically have two choices at the moment. One is restructured text which is written in Python. It's used widely in the Python community. All of the Python core documentation is written in it. A whole bunch of the libraries you use have written in restructured text, but it is a Python solution and if you're working on a project that has some Python code and maybe some Rust code or some C code or something like that, those people will probably not have encountered restructured text before, but they will probably have encountered Markdown. Markdown is a valid choice. It is simpler but also less powerful. You're making some compromises here. Both of them allow you to use tools like Sphinx for restructured text or MakeDoc for Markdown to compile a directory of Markdown or rest files into a directory of documentation that's all linked together. Both of these are supported by ReadTheDocs, so you can publish either of these documentation sites to ReadTheDocs and then not have to worry about hosting them yourself. Once we've decided and I've chosen Markdown, again, arbitrarily, we need to write a ReadMe. That's pretty much essential for any modern project. Here we have a title of the project. We have a small paragraph describing what the project does. We should have a section describing how to install the project with some sample command line code for PIP installing this project. We should have some sample code just to tell people how to use the useful code that we've published to PyPI. Once we've written this, it's nice to have this also published on PyPI, so as well as publishing on, say, Github or GitLab or wherever you're publishing your code, it would be really nice if we could make this essentially the official description of our project and we can do that. Again, this is even if you've published packages before and you used restructured text to write your ReadMe, this is now a new feature in PyPI as of about a year ago. PyPI supports Markdown directly, so you don't need to convert your Markdown to restructured text before pushing it up to PyPI. Here we're taking advantages of the fact that setuppyY file is code and not configuration by opening the ReadMe file, reading in this block of Markdown, and then we apply that string to our setup call. So we use the long description parameter just to provide this string value that we put into a variable, and then very importantly we need to tell PyPI this is Markdown and not restructured text, which we do by providing this mime type as this content type parameter at the end. I wanted to talk about dependencies. I've cut this talk down a bit, so we won't actually show any code that uses blessings, but for example if we use this terminal colouring library called blessings, this is how we would add it to our setuppyY file. So we have an install requires parameter, it's a list of these specifiers that describe the library and the versions we're prepared to accept. I will talk a little bit more about those in a moment. If we change the library dependencies or anything else, as I said, we should run our pip install minus e.command again just to reinstall the package and just make sure it actually pulls down the dependencies and that these things work together and then we should run some tests, but we don't have tests and we shouldn't just keep on opening up the REPL and randomly calling functions to make sure that things work. I would recommend you write your tests with PyTest, PyTest is awesome, but in order to write tests with PyTest again we need more dependencies, but this time we're not talking about a dependency of our library like blessings, so we're not saying this is how this is needed to run, we're saying this is a development dependency, so this is something people need to install in order to develop code with our library and in order to declare develop dev dependencies, I recommend you add them as extras into your setup py, but a lot of people here I suspect are using requirements.txt for this, if you have a setup py file I would argue you do not need a requirements.txt, you can do all of this within Python standard packaging framework and you get some advantages because again this is code not configuration. So the way this works it looks a little bit like our install requires, but it's got an extra layer of indirection, so you'll see that it's a dictionary rather than a list, but you still see that list in there as the first value, so the key is the name of your extra, so in this case we're saying dev, we will tell people that they need to install the dev extras in order to work with our project, and then after that it is just a list of dependencies, so in this case we're saying pytest above or equal to the value of 3.7, and then we can tell people how to use it, so again we update the readme, we have a section saying if you would like to help develop Hello World this is how you install the development dependency so that you can run the tests, and it looks very similar to before but you'll see we have the word dev in square brackets afterwards it's saying we're installing our current module with the dev extras, you may have used this with other packages maybe not seen how that was specified, I stole this straight from Atters which I think is why Hinnick is here, so yes if we install the extras you'll see that it installs a whole bunch of other stuff basically dependencies for pytest, so the difference between install requires and extras requires is that install requires is for production dependencies things like flask, click, numpy, pandas, and the version should be as relaxed as they possibly can be so you should be testing against multiple versions of each dependency in this way you're not locking your users into a specific version of a shared dependency so if both you and your user are using Atters ideally you need an overlap there so if you're all using like if you're using version 3 they're using version 4 they're not going to be able to use your package unless it makes some changes extras require is different it's for optional requirements for development or testing or whatever groups of extras you want to create and the version should be in my opinion as specific as possible because you're trying to get your developers up and running as quickly as possible and so creating an identical environment to yours and the other developers who have been working with the code is like that's just going to make everybody's life easier rather than trying to debug like minor variations in your development dependencies requirements.txt still has a place but I would argue it's for apps that you deploy onto machines that you control so in this case you're pinning every single production requirement to a specific version so that you're producing a well tested collection of code on a destination machine so use fixed version numbers with the double equals operator and use pip freeze to just spit out all of the things that are currently installed straight into your requirements.txt so here we write some tests I'm going to zoom through this a bit because I'm running slowly slower than I would like so but yes we run run our code now we just need to run pytest to actually test our code each time it's much easier than actually executing the code by hand it will spit out a bunch of stuff to say what it's what version of Python you're using and things like that and then it will spit out hopefully that your tests past so now what we've done so far this is this is what we've produced we've got a license file a readme file a setup file source directory with our code and a test you can obviously stick your tests in a test directory if you have more test files it's good to distribute source distributions as well as binary distributions for various reasons people can check the code before they run it they may not have access to github to access the code they may just need to verify the code before they run it when you run estist against our setup py file we actually get some warnings saying it would like some more data for some reason estist would really like to know the maintainer and maintainer email or the author email so it's told us that so we can just add those in that's three lines we add the URL of the project a link to github in this case my name and my email address excuse me so now we need to test that make sure that source distribution contains all the files that we want it to so it just when you run estist it just creates in this case of gzip table and we can use the tar command to unzip that and have a look at the stuff inside and when we have a look at it we notice that it hasn't got our license txt file or our test hello world file ideally our source distribution should contain everything that is in this snapshot of code so everything we're distributing everything that gets built into the binary distribution in order to add those missing files into our source distribution we need to write a manifest input file they are fiddly and annoying fortunately there's a tool called check manifest that does pretty much all of this for us or at least we'll get it started quickly so you can pip install it you can add it to your development dependencies if you like you run it for the first time with this create flag and then it will create you a manifest.in I recommend having a look at it it's just things like includes and excludes lines for various files that it's found in the project that it tries to make sure that everything you have in git ends up in your source distribution so it's finding these files and adding them to the manifest input file so then if we build our source distribution again we can unzip it and then we see that now just out of the box the check manifest has created a a manifest file that includes the files that we're missing so now let's publish it it's good to publish earlier rather than later if you try to perfect everything you will really never publish the package so as soon as at the point you have something useful not necessarily perfect try and get it up but apart from anything else it will register your package name on PyPI to your project so you're not letting somebody else just kind of come in in the months while you're working on your project you used to be able to register your project before you uploaded code now you need to actually upload the code in order to register a name so here we run a setup py with the b dist wheel and the s dist command and in our dist directory we'll now have our wheel file and our source distribution in order to push to PyPI you need to use twine for various reasons it separates the build step from the upload step which means that you can do these manual checks of your distribution files before you upload to PyPI otherwise it's a single command to kind of build and push up your code if you get it wrong it's going to mess things up for you so here we install twine we use a twine upload command it also uses HTTPS whereas for a while pip didn't so it's safer sorry setup source didn't if you get to PyPI the website quickly enough you will see the name of your project on the home page as the most recently updated package and that's kind of cool if you click on it you will then get to the project page you can see our readme file is essentially duplicated here there's a github link that I've just cut off at the bottom I had to change the name of the project by the way because there is a hello world package on PyPI obviously somebody has done that so there's still some more stuff that we need to do I would recommend using TOX I really am running out of time so I apologise for running through these I recommend using TOX for testing against different distributions of Python and different versions of the libraries that you depend on here we're just testing against Python 3.6, 3.7 you install TOX you have that TOX configuration file it spits out loads of output when you run TOX and hopefully at the end you get for each one of your targets you get a command succeeded and a little smiley face at the end which I always think is rather lovely here's why we use the source directory so our root directory is the directory we have I've been working in if our code was in this directory if we import hello world while running the tests it will run the code in our source directory sorry in our current directory but we don't want it to do that we want it to test installing the package and using the code from there by having a source directory you are forcing it to use the version that was just installed into the virtual environment you should also build on on clean machines in the past I've used Travis for this I am probably moving my stuff to Azure Pipelines depending on when Hinnick gets his stuff stabilized yes I won't talk about that anymore for extra credit you can add badges to your readme for code coverage for quality metrics you can manage versions with bump version that's quite nice you can test on different operating systems you can write more documentation you can always write more documentation and tests you can add a contributed section to your readme you can implement a code of conduct and there's lots that you can do but I recommend that you don't do any of the stuff that I've described in this talk so there's a project called cookie cutter that generates sets of files from templates and people have already created template projects for PyPI for Python projects so if you install cookie cutter and then you run this command to download Yonald's there's a few of them out there I quite like Yonald's it's similar to my own way of thinking of these things this cookie cutter library it will then it will download the template from github it will ask you lots of questions because it's much more flexible than all the I've given you one option for each step he offers you lots of options different testing libraries and things like that and then at the end of it you're done in theory you will probably have to go and tweak some of these files because they won't be quite the way you want but it took me five minutes to get up and running using this process and then it created all of this so you will recognise some of the stuff in here from the tutorial that we've been through there is extra stuff in there there is a Sphinx directory of documentation with just boilerplate documentation in there at the moment but just waiting for you to fill it out you copy in your code and then you're done so that took me about five minutes could have cut this talk down to the last two slides if I'd wanted instead of wasting all of your time for half an hour but hopefully give this gives you an overview of good packaging practice and all the things that you need to do to kind of build a well-rounded package there's obviously different directions you can move in but this is a really good core understanding of the things you should do for a professionally released package there are other projects for distributing libraries these days that are very interesting that don't use setup py or use it in different ways I would really recommend having a look at them if you're struggling with setup py poetry is getting a lot of mindshare at the moment I haven't really used them so I can't really recommend them I'm trying to push current most common best practice if you are interested in the slides or the code for this talk they are available on this bitly link follow me on Twitter if you have any questions feel free to grab me at the conference come to the Nexmo booth tweet at me preferably not abuse but thank you very much for coming to my talk