Welcome to Toffy TV today. I am joined by a man who scored the winner the last time Everton won made your honours. All away from the United States it is Mr Paul Rideout. Paul, thank you so much for giving up your time to have a chat with us. My pleasure, looking forward to it.  Not a day older than when you scored that goal at Wembley. I don't know, a few more wrinkles and whatever but yeah I'm trying to stay trim as you can and I see you kind of look at my sons now and they're all trim and so you try to keep up. You've got to having your mates. It was where Graven Sharp I think was moving on and they needed a striker with some experience. It wasn't too expensive and it played at that level. It was when the news came in that I was wanted by Everton. Obviously Howard Kendall was a great player in his own right. Everton, when I played against him with Salampton they were always a difficult team to play against, well supported. In that environment as a kid, nearly gone to Liverpool, you have a sense of what it really means up there. So at the end of the day I wasn't getting too many looks because Mark Haley was the number nine. It ranges so I wanted to give it a go and really excited to get the opportunity to go. It didn't work out too badly at all. Signed on the day before the first game of the season and played against Sheffield Wednesday on the Saturday in a one-month row in a game that I thought Barry Om was going to be a goal scorer midfield player because he actually scored for us that day. How wrong can you be? I think he only scored another one and that was in that Wimbledon game with Screamers. I was watching that. It was quite funny. It was a crappy goal, wasn't it? When he scored he makes it work for you well. Oh, not bad, not bad. But it wasn't the greatest Everton side obviously at the time. We were struggling a bit. How did you find it when you came in? I mean your second game for us. We beat United 3-0 at Old Trafford if you remember. That was obviously the first Premier League season. It was a tough season obviously. It wasn't a good dressing room because there was a lot of older players that were about to move on and there was some new players that had come on. People weren't getting on well with each other so the dressing room wasn't the best to be fair. Obviously the pressure of getting results at a club like Everton, which it always is every year, I watch even now. If you're halfway in the table it's not good enough, especially with the money they're spending. That said, I knew it would be tough. I said it before. I was in a school. The first season outside of that, everything else I really enjoyed. All I talk about is Everton now. I still love Southampton. It's my home club. Everton is where I really feel a relationship with.  It is anytime. They're proud people. It costs a lot of money to come to games. If players aren't performing, it is what it is. You've got to expect some of that. At the end of the day, you can't control everything. You have the good times, the bad times. When you get the good ones, like the FA Cup, scoring at Ipswich, staying up, that was one of my best moments, probably better than the FA Cup just because of the importance of it. The win-win game, but you don't want too many of those days. No, we really don't. We beat Southampton at our own 1-0. I remember driving home from my dad's car going home from the game and he'd resigned. We were puzzled as fans, but did you see that coming as a player? Or was it a shock to you? You never knew what he was thinking. He was such a lovely guy. He really was. More of a friend to the team than a man to it. I don't think the discipline was there enough. The fitness, when we moved on a couple of years on in Dogs of War, we were so fit and sharp. We all knew our jobs, but there we were almost playing off the cuff. It wasn't so structured. What I can't say with Ken was that I loved being around him. He made you feel wanted and he made you feel good. He was a lovely man. Coach-wise, I don't know all the time yet, he did a fantastic job for a certain time. I think there was much better players before than there were at the time I was there, including myself. When I'm looking at Andy Gray, for instance, I touched Andy Gray. He was fabulous. He was one of those I learned how to head the ball from. He wasn't the biggest guy, but his time and his greatness was fabulous. That's what I learned from. It was that type of time. We had a change and things move on. Walker, if you look at even Norwich, you might disagree. A lot of Norwich fans might, but I thought he got that job. He walked into that job when they were a top team anyway. He had a lot of good players at Norwich. It's nothing to do with a personal side. He didn't fancy me, but I would never say that about anyone. There just wasn't structure. There just wasn't a work ethic that you could say to yourself. We've got a chance to get out of here and push on and do something. It just wasn't there. As time went on, fans realised that. You could see if you asked any player, a majority of players, they would say the same thing. We needed something there that we could work off. You can't play off the cuff at that level. It just doesn't work. Structure is a be all and end all. It can be different. We were dogs of war and then you look at Man City. They played terrific soccer. He's planned that. He worked on that. He made those players over time understand that this is how it works. You can do your individual stuff. It was never going to work. Again, I thought I would be moved on and wouldn't be a part of it. It came to such a situation that they decided to make a change, thankfully, at the right time. We still don't have time to change it and turn it around. Obviously, you know what happened after. Before we get on to Joe Royal, I've asked Neville Southall to say all you need to know about Mike Walker. He's not as... You know what Nebs like, Paul? Nebs are not back with him coming forward. He's honest. He's honest. He's as it is. That Wimbledon game, you touched on it before, obviously. Things have happened and we'd found ourselves needing to win that game. The final game of the season and playing them when they were a side, weren't they? The tough side, big team, tough team. What was that like to be involved in? You're at the club and all of a sudden the park ends being knocked down and people are in trees and stands getting built and everything was crazy, wasn't it? It was. It started with the day of the game of driving a bus from the training grounds, parking our cars there and not at the stadium because of the aftermath if we did go down. Then you get there and the tension is not just the game, the whole build up to it, the whole week of it. It was all you could think of. You couldn't sleep. You didn't want to be a part of that team that would get ready to get it anyway. You get in there and then you hear, this is the biggest roar and I've been in some big games, the Celtic Rangers and all that sort of thing. But that roar, that particular day, never stopped, never dropped, two nil down after I think about 12 minutes and you're going. Again, I looked at the tape and I saw a couple of, Holdsworth missed a couple of absolute sitters. You're two nil down and we get a cracking goal to bring us back into the game and you're in there at halftime and we're down and you've got a lot to do. And Wimbledon don't care about anyone. If it's Liverpool, Man United, they don't care. They come and play the same way and they never give up. But no, I mean to be fair at halftime, the dressing room wasn't what you'd think it would be screaming and shouting. It was pretty calm. Everyone kind of understood that we had to go out there and give it everything. Again, the only way you really could do that is by the way that the fans treated us. They never once dropped their standard of keeping on our side. It was amazing. Two nil down. You had a deaf silence for about five seconds and that roar came back in. I've never heard anything like it and that's what kept us going and that's what got us the result I believe. Unbelievable, unbelievable. Never want to experience that again. Thank you. It wasn't great but we come out the other side of it. Joe Royal and Willie Donoughy come in and we play Liverpool at the bottom of the league. We play Liverpool who are third in the table. We tend to go top, I think, that night in there. Big Duncan, and then you. Get us three massive, massive points into Joe Royal's opening game. Yeah, that was in the way that they came in. We went into the gym and Willie Donoughy said this is how we're going to do it. And if there's anyone here that doesn't want to do it, off you go. Obviously no one did but it was just work right and organisation, everyone understanding this is your job. No more right now. This is what we need and we worked on it, worked on that. If you were coming to watch training you'd go, Jesus, I'd hate to watch that. I hate to be involved in that every day but it was what was needed. I think that when we went out to the Liverpool game it was probably the ideal game. If you can't get up for that one, you can't get up for any game. It was great atmosphere again and Liverpool fans were sending all sorts of messages out there at that time. Free was a run down to the game and it was a pretty slow start by us. They had a couple of opportunities and we would have looked a little bit nervous but we built ourselves in the game and Duncan himself a great, great goal. That's what he was really strong at, bullying people. They gave us an opportunity to work on being 1-0 up but like you said earlier that we were throwing those things away. But now we had a plan with the defensive side of it that we'd stay strong and stay solid and everyone knew their job and stuck to it. We had a lot of workers out there that were willing to do that. Duncan was a much, much better player than I was in the respect that he had some serious pace. He had a fantastic touch for a big man and he had great, great vision. He had a lot going for him and it made my job easier. I felt like I was a good technical tactical player that could pick up the bits and bring in the midfield and use the white guys. But anyway, all said and done, I got the opportunity to come on at halftime and it gave me enough time to get into the game. Obviously the goal I felt was not just important but one of my best goals because it was so awkward. And I didn't think I would get there in the way that I came in and bullied it. I went through someone's legs so it was fortunate but at the end of the day I felt it was a really good goal. But that did so much for your confidence. But that said the next game I started and that confidence from that to that I just felt like I could do anything because I had Duncan up there. I had a midfield that was so hard working, so organised and so tough. I wouldn't want to play against Barry and Joe. They go through anything. And we had a solid defence all of a sudden that we're working on certain things that Willie had instructed to do. We didn't lead gaps in the back end and that little run we went on of beating some really good teams at the time. It was pretty impressive. So that you grow on that and obviously with everything else with it, the feel good factor, the fans were back again and enjoying their soccer. It just felt a much, much better place. It was incredible to watch the team. The West Ham win, which was the last home game walker had, was a navy win. You were throwing yourselves in the way of shots. We should have beaten them by more. We had the answers, but then they missed the couple at the end and it was navy. The difference between that team and the team that took the pitch against Liverpool or rather finished against Liverpool at night, we were a good side. It was like night and day, the way, right, the ethic. Was it something in terms of the fitness, did Joe and Willie notice that immediately? Was this like listening you aren't fit enough? Did you feel out as players at the time? Or are you just in that zone of this is what we do? And you're thinking I'm a pro, I'm fit? Yeah, you do. You do and then you have a wake up call. But I think along with the fitness, it wasn't just that it was a mental side of it. If you're, like you said, if you're not on board and I don't mean just your soccer or just wanting to be here or be in the team. It's like you have to throw yourself into this. Now, every pro should do that every time. But we weren't. And when you're not winning the confidence, it's hard. There's times when I've been on the field in some games, not particularly with Everton, but in general. As an older player, you get scared of missing. Whereas a younger kid, like I said earlier, you don't have that fear because you know that. And the next one I come on, I don't care the crowd, I don't even care. And that's what that was. It was a whole confidence factor of what they brought. And you can be the best player in the world. Everyone has a downside of bad time going through that. I just felt that you look in the dressing room, everyone's talking, everyone's buzzing, everyone is looking forward to training there. And looking forward to getting on the field. We were dreading getting on the field and thinking, I don't want to let the fans down again. And it doesn't matter what level you play that, you go through that process. But Joe was so good to go into his office, even if it was nothing to do with anything. You're so welcoming.  And I wouldn't go any further. I just enjoyed that time with those guys. Absolutely outstanding. You scored six days after that. You scored a winner at Chelsea. Can you believe that we still haven't won there in the league since your header? I know I keep getting mentions again. I keep living off those little things. It's not a good thing. That's the problem. I'm one with James. But yeah, that's crazy. I actually scored another one in that game. But there was nothing wrong with it. I was mystified when he called it off. That was a good choice of team as well. We just went out there and we just bumped him. They didn't have an answer to what we had, because I think we just surprised everyone without giving them any kind of time. That was a lot of good, solid players in midfield. It was excellent as far as tactical and technical. But work rate, they were over the top a little bit and we took advantage. And a terrific win that one. For you, playing with Duncan, because obviously there was so many times in your career where you were the target. Even though Duncan is a big lad, but you were the target. Was that something? Did you have to learn to play with a big man? Or were you just like, great, he's going to take the knocks and I can try to pick up the bits? Did they make your job easier or what do you prefer being that focal point or not? It did. I love playing with Tony Cotty as well. He's a good striker to work off. He's probably clever, very intelligent. But what it did for me was, I still won my fair share ahead of us, but I didn't make all the knocks all the time. You see defenders at times were scared of Duncan because he wasn't scared of putting the elbow in him. And he was a clever player in that way. Even in training, I wouldn't want to come up against him. He just knows how to use that body well like Shearer did when I was at Southampton. He was so clever. Yeah, it totally gave me more energy. I felt that I was, in this day and age, I would have been an A. I wasn't quick and Duncan on the run was quick. So, I thought I had nothing in my game to be able to work off that and still do some of that and then get into the box a little bit more often. So, maybe that, therefore, I was getting more goals. If you scored your first cup goal in the five-nil against Norwich, we'd had a... We beat Derby County in the third round. It wasn't a great game. Hinscliff scored late on. And then Bristol City away. I mean, I still don't know today. Still, 25 years later, how we got through that game because we got battered in me. And then Matty, we had Matty axing him before Christmas and he was like, I don't know what happened that day. I just did it with me swinger and it goes into the net for real. I apologise to the fans after the game. That was, yeah, we were shocking, shocking, these straights. And then that's what you need though, a little bit of luck that you sometimes don't get. But yeah, that was a big, great escape that was. It could never have been. No, but name on the cup stuff, wasn't it? Yeah, but after that, it wasn't an easier run all the way through to the final. All good teams. So we earned that right, I think, in the way that we were playing. That is where we got too big for our boots, but we learnt from that in the Derby game and the next one. Derby game was tough. It wasn't an easy game. But then, like you say, the Newcastle game was a good game. They were a side, weren't they? They were a good side. They were. And a great goal to win the game. You get to top them. I just fell on that day, and I think the group did, that top them for the team they were individually. I mean, great forwards. Their defence was their weakness. We just fell and could exploit that because they weren't fabulous in the air. Top them was one of my more favourite teams to play against because they played it the right way. But they never seemed to have central defenders that were tough guys. They were just technical smart players, good soccer players, but not particularly good in the air. So I always felt that I would score against them and just have feelings against certain teams. It was unusual. What was it like coming out of Elin Row that day? Because the way they distributed the tickets, they gave spares that big stand. But we had the three sides. We were being a fan in that ground that day. It just felt like a home game. It was just blue everywhere. What was it like for the players coming out onto Elin Row that day? Well, it was fabulous for us because you look round and you realise, where are the Tottenham fans? And they probably got that big stand. We never heard of it. The fact is we never gave them a chance to start singing. We got first goal and from then onwards we never looked back. The game was never a game, I don't think. We just stunned them. I think at that time everyone wanted the Tottenham man United final. We were having none of that and our fans were again exceptional. To be fair, how did we get Elin Row? Because that's not far from us compared to Tottenham. Everything was on our side. Again, in itself, that was the strange game. Matt Jackson was scoring a header, putting us up. We needed to chat about you in this. We needed to chat about you because first of all, you almost make it 2-0 and Graham Shew put in your shot. It's a good shave from the keeper. Graham Shew rolls it in for 2-0, which is incredible. I underestimated that finish. Great finish, but what happens? Because you went down injured. We've heard all the stories from Tio, from Ammo, and the press that he subbed himself on. You were getting treatments on the far side. I said to the physio, he's not effing bringing him on, is he? I'm going back on. Tell him. It was just one of those things where all the way back, as I was walking around, fans gave me a fantastic reception. I was effing and blind and I was down. How did this happen? Once you get on then, then he comes on his course straight away. What can you say? That guy was such a happy guy. He's such a cool guy. He's a top-young player at the time. In essence, it was fortunate for me because my knee was damaged. In a way, he did me a favour because I probably would not have made the fine. I was struggling to make it as it was through the treatment. He came on, he lit it up. It was a different type of player that they couldn't handle. It worked out for everyone. It's just a madman. Obviously, it is lucky that he came on. Obviously, because we wouldn't have wanted to lose you for that.  I suppose his pace on the counter attacked near the end, probably played into it. That was the only goal we conceded in the whole cup run. It wasn't even a penalty. Sharon bought the penalty. He was very clever in both the pen. He was not a good player anyway. I don't think for any minute in that game that we felt we were going to lose that. We were so relaxed. It was just a reward for getting out of the mess we were in. This was a bonus all the way through. Other than the Bristol City game, a lot of us have talked about it is that it just felt comfortable going through in every game, the Newcastle game. It might have been different in Newcastle, but playing in our home ground, you just didn't feel like you were going to lose that game. Going to Tottenham and then even the final. It's our time. It's such a belief of the team. You mentioned it right before, but we were on that really good run. Remember, we had three own games on the run. We had Southampton, Wimbledon and Chelsea in a week, all a goodness, and it was when one drove on the sand. More or less we were okay. We drew all three of them. Two nil nils and a three three, which meant we ended with two away games. We went up to Ipswich, which was a real tricky game, because they were difficult to go to Portman Road, but you come up with a massive goal at night, didn't you, to secure our status? Actually, I think that's when the team realised, let's just get it done. Let's give ourselves a game to enjoy before the final so that we can work on stuff. Ipswich was our weakest opponent. They weren't doing particularly well. But given how far that is away, on a weekday peing down the rain, the fans are phenomenal. It was just excellent. We never felt like we were going to lose that game. There's one nil, but it should have been more than that. After that game was where you just say, hey, we can just enjoy these next two games. Even though it's the last home game in the season to thank the fans, and then, obviously, Wembley, you're going to enjoy it. If you've got that last game, you're still worried about relegation, it's a big issue. We got that out of the way that the goal was kind of a fluky goal, but it was an important one. That was actually a better feel than the FA Cup because of the importance. Again, it just is. You can win a cup, but if it's not league, then what does that all mean anyway? Going home on that trip, we had a great trip back, enjoyed it, and you're just preparing for the final. It was such a huge win. You write it, we still had Coventry away, everything was done. Everyone was like, forget the Coventry game, it's Wembley now. Obviously, in that little period, we'd lost Duncan as well. We had Duncan struggling as well. I think Duncan came on at Coventry the last half an hour or whatever. We ended up drawing, but then you were saying you were struggling in the week of the Cup final? Yeah, just with pain on my ACL inside. I felt I'd be OK for the final, but it was one where I couldn't play the Coventry game, and Joe was good enough to give me a rest. Obviously, that shows the importance of winning the S, which gave me an opportunity to do that. What I'll always think about is that when you look at the final, and then you look at Duncan's on the bench, and I'm stunned, that amazes me, because it's probably not right, because he's far superior than I was. But I think that I was in good scoring form, whereas Duncan wasn't scoring as many. It was a good season for me, and I just felt like I'd scored every game. I think that was where Joe Royal made that decision, I think, about the kind of premises that I would deliver. I actually came off at half-time, and Duncan came on and did a great job for us. But I was really surprised that I was starting that game, so I'm thankful for it, and things worked out, so it was good. You ended the season with 14 goals in 29 games. You were our top scorer, one and two in the Premier League. That was just Premier League goals, by the way. I think it was something like 17, 18 or whatever overall.  I loved Duncan, but he was never scoring that kind of goal. That many goals that you were getting, he was more the focal point. For me, it was a no-brainer that you were starting. I remember being in the crowd, and it was whether Duncan was going to make it on the bench, and him and Amacati were on the bench, and that gave everyone a boost. I've said this to Matt Jackson, it was so weird. We were on. We were a team that had you scored the goal only 10 days earlier. But that second half of the season, we were on a great run. We beat United at home and things like that. I was that confident going to Wembley against a team that had lost the league on the last day of the season. A week before we were going for the double. When Graham Stewart hit the bar, I didn't follow the ball. I just looked at the net, expecting the ball to hit the net. It hit the net. I'm expecting it anyway. I didn't realise you'd scored the first, because obviously, Bedlam breaks out. I've just seen the ball hit the net. Running everywhere, tears in my eyes. That team had just created that for our fans. We just expected to beat the team. Now, when I think about it, how did I expect us to just beat United then? We were 15th in the league and they were second, but that's how good a run that team had turned in in that second half of the season. Again, that goes down to management. Obviously, the players, you do your part, but without the leadership, it's very difficult. That's why the importance of having a solid manager that not just understands the game. That's why Pep, I love him, because he's such a manager, he plays manager, but he's so smart with everything. There's not many of them around. I won't mention Klopp, but at the end of the day, I think we've got one now with Angelotti. I think he's with the experience. I played against him in Italy. A fabulous player. I just think that it was a great manager to bring in at this time with such a strong squad. I think he can bring it together. That's what belief is. That's why the wing games, you have that belief, not just ability. You look at Man United, they've got a lot of ability, but they have no concept of team play in my eyes right now. They've improved, but they're still there and over there, and that's what we were. We still lost games, but we knew why. Sometimes it was just pure good finish, and sometimes it was... We didn't finish our chances, but there was a reason for it. Going to the Cup final and playing in front of, again, it felt like a home game. It really did. Whether Evertonians allowed it or not, I don't know, but you couldn't hear anything else but them. When we were out there, if you look at the entire game, we were outplayed, but then they had some great plays at the time. They had a solid defence. They didn't concede much, they were second in the league, but they had something really to come out. They didn't want to come out empty handed. That was a danger for us. We were second, we ran out and enjoyed it, so there wasn't pressure in the way that it would normally be, like the last game you've got to stay up. So it wasn't there, and you didn't feel it. You just enjoyed the occasion, and I can remember every minute of it, but Man United, the best part for me, is you understand how much it means to Man United fans and their team, is when we went into the bar afterwards, and we saw six or seven of them over there, silence, not a word to each other, looking over at our table, and we're just loving it, and they're going, I think you just lose to that. You could see it in their mind, discussed, but that's soccer, and I don't think they respected us enough. And again, Neville South, we had a phenomenal game, and Dave Watson was unreal in the middle there. It really was a defensive play, rather than everyone remembers the goal, but the way that we defended as a group was what was brought in at the start of the season, and it stood us well over that whole season, right to the last game, and that was a difference, but at times to win a cup, you need luck, you just do, and we certainly had it on the day, but what a great day. Was it a surprise when Joe left, because it surprised us as fans? Yeah, you don't know what goes on behind the scenes, but that really did surprise me, and it's a bit sad to be honest, how it happened in the end, but that's soccer. When you look at the last of 10 years of Everton, and how many coaches have moved on, I just hope this is one where we have that stability of four or five years, and really, if you look at all the best teams, the teams that have done really well over time, the managers have been there for a long time. If you look at Liverpool man United, with Ferguson and Man City, he's been there four years, five years now, they're all his players, and it does take that much of time. People don't get that much time now, but I just believe that we have the right person now, and that it is what it is, so fingers crossed it works out. So your last appearance for Everton, was obviously Dave Watson took over for a couple of games, and you weren't involved, and I don't know whether you were injured, or you just weren't involved, but then you were at the playing centre midfield against Tottenham at home, and you were absolutely brilliant, I think you were a man of the match, we won the game 1-0, Gary Speeds scored early on, and then you left on a motorbike, and that was the last we saw here.  Well, the game itself, I think we had an injury in midfield, and I just said I could do a job there, because I played with Glasgow Rangers as a centre half in a couple of games, I got a man of the match there, because it's Scotland, and at the time, Rangers were magnificent, they could beat anyone. I don't mean that about Scottish Soccer, it was good, it was a good lead, but then I told the coach that I felt I could do a good job, I was fit and I could get around the field, and I, again, I thought I had decent vision, so it worked out really well, it was a good game, and it was a tight game, but Speedy came in with a great goal, like he did, a fabulous player, and it just worked out for us. It was a really, really enjoyable way to leave the club, kind of almost taking him out of danger, but I was supposed to, Dave Watson told me, you're staying for the Liverpool game, and I said I can, otherwise this deal is going to go sideways. I said I'm begging you to let me go, and in the end he said, you can go, which is cool of him, and obviously getting out of the dressing room, and then someone told me I had so much time to get to the plane, and he said you ain't going to make it by car, and I've got a buddy that, we've got a motorbike ready for you, so ran out of there, ran out of the dressing room, jumped on the bike, and trying to beat all the traffic, it was crazy, and people doubled taking so they could see where it was. It's like, what the hell's going on there?   Kept us off, we won the game, and you get off to the airport on a motorbike like Jason Bourne, not a bad way to finish, not a bad way to finish. I mean, enjoy that 108 degrees heat that you're having there while we switch our heaters on over there. Listen Paul, thank you so much for taking the time. I really enjoyed talking to you, thanks so much, appreciate it. It was outstanding. Best wishes on to Jordan, because we speak to Jordan through Twitter and all that, and obviously you miss us for shortening it as well, outstanding Caroline there, so thanks, stay safe, and hopefully you'll get to Tiatr here again at some stage in the future. Cheers, thanks a lot, take care. The brilliant Paul Ride out there scored the goal the last time we won a major trophy. Thanks so much Paul for taking the time out. Make sure you give this video a thumb's up, subscribe if you haven't. If you want to see the full uncuff version of this, get over and join us on Patreon. See you later.


Welcome to NTN Nightly. I am Jesse Leance. This edition stops stories. Prime Minister Hon. Alan Chastney prepares to lay the estimates of revenue and expenditure amid the challenges of COVID-19. The OECS launches a series of activities celebrating its 40th anniversary and the Ministry of Health and Wellness to begin vaccinating homebound individuals. Prime Minister and Minister for Finance, Economic Growth, Job Creation, External Affairs and the Public Service Hon. Alan Chastney will on Tuesday 16 March 2021 present St. Lucia's estimates of revenue and expenditure for 2021-2022. This will follow the opening of the 6th session of the 11th Parliament on Tuesday morning where members of the House of Assembly will meet at 10am and members of the Senate will meet at 10.30am after which both houses will be in joint session to receive the thrown speech from His Excellency, the Governor-General, Sir Emmanuel Neville Snack. The Standing Finance Committee will meet in closed session at 2pm to review the estimates of revenue and expenditure, following which at 4pm the Prime Minister will lay the estimates before the House of Assembly. The Prime Minister says the estimates of revenue and expenditure for the subsequent appropriation bill are being presented at a difficult time. However, government has been able to maneuver the tough environment brought on by COVID-19. Government, he noted, has not defaulted on loans, continues to pay salaries and meet other obligations. Nevertheless, there is tremendous uncertainty about how quickly the economy will recover from the impact of COVID-19. Government has had to give a lot of concessions to small businesses, medium-sized businesses and large businesses to offset the losses. And so we've absorbed those losses by not charging taxes on a lot of issues. So what government's final revenue position is going to be is still very much up in the air. But what we can say is that a continuation of building a new St. Lucia, a continuation of fixing the infrastructure, a continuation of making ourselves competitive. I mean, all this week we were talking about the Headquarters Act, the residency program, expediting our financial sector, the orange economy. We have a tremendous amount of confidence in it. So there is an allocation for events. We're hoping that we can see at least one event come back this budgetary year. But we feel very strong that Carnival will come back strong next year. So there will be an event probably to celebrate something as well as being a launching pad to promote our events for 2022. Prime Minister speaking there during the GIS COVID-19 year in review production. Hon. Shaste emphasized government's determination to build a new St. Lucia as he noted the advancing of several projects including home porting in Viewfort, the National Health Insurance, St. Jude's Hospital and the expansion of the social safety net in providing assistance to the vulnerable. The social net program we're going to be putting in which will be much broader than it has been before. So the continuation of infrastructural building and infrastructural is not just about roads and buildings but it's about institutional strengthening, about policy strengthening. Our police officers, the police headquarters, the continuation of the digital cameras, the CSI unit, the crime scenes investigation unit that we're going to be opening up. We just got new engines for the Marine Police going to the border control. So as I said to you this year really is a continuation of a lot of the things that we've already started and despite COVID is to give solutions the assurance that we have made the physical space available and the human resources available and continue to prioritize those programs. The debate on the estimates of expenditure will begin on Wednesday, March 17th, 2021 from 9am. Please note that in keeping with the protocols established by the authorities for the management of the COVID-19 crisis members of the public will not be allowed in the chamber gallery during the sittings. The public can view the live proceedings on the National Television Network Channel 122 that is NTN as well as the government of St. Lucia Facebook and the YouTube platform. The Ministry of Health and Wellness is looking to make good on gains made over the last month in the fight against COVID-19. The national vaccination campaign is in full swing and the curve of infection from the third wave has flattened. There is a renewed focus to avoid another surge following a holiday observance. As of 13th March, 18,488 residents in St. Lucia have received their first dose of the Oxford AstraZeneca vaccine. Chief Medical Officer Dr. Sharon Belmar George is elated by the progress made in the journey to herd immunity, however stressed the importance of continued adherence to the protocols. COVID-19 continues to loom over the population following the island's third wave at the start of 2021. Active cases registered at 14th March stood at 203. We still have too many active cases within the community. We still have over 100 active cases within the community and a lot of our risks for introduction are still there. So the vaccine it really provides a light in the tunnel and the end of the tunnel. It is one of the most cost effective public health measures. So we extremely grateful to have it available to the public and they will continue to appeal to the public like Nishabati said to take the vaccine. However, we have received just the first. Most of our persons who have taken the vaccine is only the first dose that they have received so far. So it is important and I will continue to appeal to the public that we continue to maintain the measures that are in place. Dr. George was speaking during the COVID-19 year in review broadcast of the NTN on the weekend. In retrospect, she admits that the post-holiday case of Surgeon January might have been avoided had it been managed differently. When I think back, we should have limited the number of persons who came in or keep everyone within an institutional quarantine because persons coming in during the Christmas season which traditionally we know is a time of festivities where there's a lot of family activity happening, a lot of community activity happening and it did contribute significantly to our third wave which we saw. To ask persons to come in realistically during the Christmas time and to think that they were going to stay home for 14 days, it was not realistic. Lessons for the future. Officials are taking precautions for the next holiday observance. An extended curfew is among restrictions for the upcoming Easter weekend. From 2nd to 5th April, it will run from 7pm to 4am daily. Commissioner of Police Milton Daisy reminds the public. Coming into the Easter weekend, the curfew will be back to 7pm and we are expecting that everyone would observe the protocols and one of the protocols being that there should be no socializing, no beach parties, no river lines, no dances and so on. We love musical meetings. We are not ready for that yet and social events are not approved. Pascal's events are not approved. So we want a level of responsible behavior. The police cannot be everywhere and in every community. So we really want at the level of the community that we make responsible choices for our safety at this point. Meanwhile, the COVID-19 vaccination drive will cater to individuals who are homebound due to age or immobility. Therefore, the Ministry of Health and Wellness will not be administering the vaccine at vaccination sites around the island. Appointments must be made for these individuals at their nearest community wellness center for vaccination at home. The Organization of Eastern Caribbean States OECS Commission hosted the official launch of the 40th anniversary of the OECS, presenting planned activities to commemorate the June 18, 2021 milestone. More from Herma de Mac. The virtual launch included addresses by the heads of the organization, the unveiling of the theme, onward with integration for progress and sustainability, a logo to commemorate the momentous milestone, and a calendar of activities which inculcates the areas of education, youth, theater and community. Addressing the launch, Prime Minister of Dom Mika and Chairman of the OECS, Hon. Roosevelt Scarrett, highlights some of the achievements of the organization since its inception in 1981. The Eastern Caribbean Dollar, which for the last 37 years has maintained a fixed and immutable exchange rate with the U.S. Dollar, and is now the strongest and most stable currency in the Caribbean. The freedom of movement of people that allows the citizens of member states to travel without hindrance or restriction to work, to play, to study, to visit friends or to relocate, to follow carnival or to follow a crusade. With those rights came contingent rights that empower our families to relocate with us and enable our children to attend schools and to obtain scholarships. Also speaking at the launch was Dr. Didacus Jules, Director-General of the OECS. He explains the requirements for future success of the organization. Importantly, our heads of government need to maintain a very strong solidarity among themselves, and we've seen this happen. Every time we have faced a crisis, our heads have rallied from the hurricanes to the pandemic. We've seen that solidarity being expressed. Dr. Jules says listening to the needs of the people of member states is critical. It is very important that the OECS communications involve not just sending messages out, but listening to the voices and not just listening in events like this, but also having structured mechanisms for getting feedback, for receiving queries, for soliciting suggestions and recommendations. And right now we have over 40 virtual working groups involving persons in the private sector, civil society, in government ministries, across different portfolios and sectors, engaging in discussions on different aspects of regional integration. Governor of the Eastern Caribbean Central Bank, ECCB Timothy Antoine, congratulates the OECS on 40 years of exemplary service. At this juncture, the OECS Commission and the Eastern Caribbean Central Bank enjoy excellent relations. Our most recent collaboration was in supporting the vaccine rollout in our member countries. So, on behalf of Team ECCB, and on my own behalf, I extend again heartiest congratulations and wish you every success for your next 40 years and beyond. One exciting component of the OECS's 40th anniversary is OECS Gourmet, a Chakrapesh production which will be aired on the 12th and 30th of June on the social media pages of the OECS. From the Government Information Service, Hermione Mark reporting. Still on the regional front, the Caracom Council for Human and Social Development, COSOD, has committed to working with the Caribbean Examinations Council CXC to preserve the integrity and validity of the regional examinations that are scheduled for the period June to July 2021. Tucson-Kind English Francis of Caracom News Time reports. The COSOD met on February 26th and March 1st and reached several agreements on the administration of the 2021 regional examinations, including that the examinations will be held in their original formats. That is the Caribbean Advanced Proficiency Examination Cape, the Caribbean Secondary Education Certificate CSEC, Papers 1 and 2 and 3 for private candidates and for the Caribbean Certificate of Secondary Level Competence, that is C-C-L-S-C, Papers 1 and 2. Chairman of the COSOD, the Honorable Jeffrey Lloyd, Minister of Education of the Bahamas, told a post-meeting media briefing that after careful analysis and with consideration for the well-being of the students, the COSOD also extended the deadline by which students should signal their intention to defer the sitting of the exams. Candidates now have until the 1st of May to do so. The COSOD expresses its appreciation to the CSEC for its efforts to ensure the smooth transition and matriculation of the region's youth to the world of work and post-secondary or tertiary education. Recognizing the impact of the 19th pandemic on equitable access and the continuity of learning in their deliberations made every effort to advocate for concessions to be provided. Thus, the ministers endorsed the recommendation to provide additional time for in-person teaching and learning to complete the school-based assessment, close teaching gaps and increase the chances of successful outcomes. COSOD agreed that CSEC will share with the Ministries of Education the broad topics for Paper 2 five weeks before the start of the examinations. However, Dr. Wayne Wesley, Registrar and Chief Executive Officer of the CSEC has cautioned it is important for teachers and students to complete the entire syllabus for their respective examinations. As it relates to the sharing of topics, while CSEC will submit to all Ministries of Education the broad topics to be assessed on Papers 2 five weeks in advance of the administration of the examinations, which means on May 10th that listing of topics will be available to the Ministries of Education. It must be said and indicated all candidates must be reminded that all topics are covered in Paper 1 for all subjects. In addition, there are certain subjects so which the details are not readily specifically available in the syllabus we will make sure that information is provided. Of interest to young people seeking entrance into the University of the West Indies, the UWI University will consider their Year 1 CAPE results. The COSOD agreed to extend the matriculation waiver UWI introduced in 2020 to allow it to consider CAPE candidates for admission based on Year 1 results given that Year 2 results were released after the start of the academic year. Minister Lloyd said the Carbon Primary Exit Assessment is scheduled for the 27th and 28th of May and the results can be expected by the 15th of June. The period for the online training and assessment quality audits for Carbon Vocational Qualification is May 2 August 2021 based on agreements with individual participating territories and Ministries of Education will receive the results in the last week of September. The Ministry of Infrastructure, Ports, Energy and Labor on March 13, 2021 commenced the VG Roads Reconstruction Project presenting a multiplicity of benefits to both residents and road users. The scope of works includes complete reconstruction of the road surface with proper drainage and residents access. This project is funded through the Road Improvement and Maintenance Program financed by the Taiwanese government. The project duration is four months' weather permitting. This is NTN Nightly. Up next, we have Primus Hutchinson with the NTN novella que all. 2020 is a year that we do not want to revisit. It was tough, frustrating and scary. From family to friends to co-workers and clients, someone we know has been affected one way or another from this pandemic. Let us take responsibility for ourselves and the lives of those around us. On an island of 182,790 people, one life lost rocks the entire nation. So let us overcome this together. Wear a mask. Wash your hands for 20 seconds with soap and water. Stay home and social distance. Share accurate information. Sanitize your hands frequently. Get registered for your COVID-19 vaccine today. Let us bring our society, economy and health back to normal. Together. Together. Together. Together. Together we can win this war. This message is brought to you by the management and staff of Invest St. Lucia. Welcome back. We now join Primus Hutchinson for the NTN novella que all. Monsieur Uter, Monsieur Madame Département, que n'est-ce que sabidité pour la formation en gouvernement cette LEC GIS et la télévision nationale PIA NTN pour le nouveau accueil, pour le premier Hutchinson. Sezième session des 11 ans du Parlement qui est pour un coup mardi, les 16 en mars 2021. C'est quand même un conseil qui est assamblé à 10 ans. Bon matin. Et après ça, les représentatives, tous les défaçades, c'est qu'à ce parlement, avec les mames Sénétla, qu'il a sablé à 2 ans session ensemble pour recevoir commission et le message, le gouvernement général, cette LEC, c'est Emmanuel Nevel-Snek. Comité des Affaires Finances, qu'il a sablé à 2 ans session par Corrillon, à 10 après-midi. C'est les 16 en mars 2021 pour viser, ESS, BGA, pour 2022. Qu'à y concentrate, qu'à y change en session, les 16 en mars 2021 à 4 après-midi, qu'on ne serait même tous les dufaçades qu'à e tromper en discussion et qu'on n'a pas constater présentation BGA. Comité des Affaires Finances qu'à y présenter au apport, c'est au apport aussi, d'une session. On se rend qu'à y concentrate, c'est pour adopter au apport, finance to submit for the financial year 2021 to sell a total of 4,688,600,900 dollars. In consideration of these protocols that I have established by the authority of Maladi Coroner, I would like to ask the public to give me permission to attend sessions in Galway and Cayepalema as usual. I would like to hold sessions in front of the public as well as the NTN television, as well as the YouTube and Facebook government pages. Speaking of that, I would like to ask the Minister of Finance to give me permission to attend sessions for the year 2021 to sell a total of 4,688,600 dollars to sell a total of 4,688,600 dollars to sell a total of 4,688,600 dollars to sell a total of 5,600,000 dollars to sell a total of 4,688,600 dollars to sell a total of 5,600,000 dollars to sell a total of 4,600,000 dollars to sell a total of 5,600,000. As a person from the US Mercobjects Group, the different sourcesidelight form that we talked destroy the and never be martyred. As a result of the COVID-19 pandemic, the public will be allowed to attend the session, which will be held, as we are used to it. You can watch it on NtN TV, on Facebook, on government pages, and on YouTube. This is Denny Nouvela. I'm going to watch the diverse reservation, so we don't have to do anything about it. We have to do business, operation business, all business, we have to close at 8 o'clock. We have to do supermarket, supermarket, restaurant, we have to close at 8 o'clock to make it easier for us to close at 9 o'clock. We have to do it together. After this, we have to do it together. We have to limit the number of people who are in contact, and observe all the protocols, to preserve that a lot. We have to do activities, we have to do everything. For the religious institutions, we have to observe these protocols, and we have to do everything. We have to keep track of every religious issue. We have to do everything. Men and women, we have to do everything. We have to do everything. We have to do everything. activity. So um um supermarket. Minimum petrol, station petrol, government petrol restaurants and cabarets. If you want to edit a sale, we can do that. It's a good idea for us to help each other. We can buy and do it ourselves. The restaurants that we are working on, we need to go through the process, if we are able to observe the protocol, we need to go through the process so that we can observe the protocol and see what we can do with it. I'm very pleased that we are able to work together. We want to give information, We want to continue the call. The call will be delivered and the operation will be completed. We also want to thank you for your time. I would like to thank you for your time. I would like to invite you to join me in the next episode of the show. We will be back with a new episode. We will see you in the next episode. On the St. Lucia Government Facebook page or our YouTube channel. I'm Jesse Leon, signing off for now. Do stay tuned for more NTN programming. Goodbye.


Welcome everyone to this session of free based DSL for distributed compute engines by JoyDeploy. We are glad that JoyDeep can join us today. So without further delay over to you JoyDeep. Thank you everybody for joining this talk on free based DSLs for distributed compute engines. So I'll be talking about some of the work that we have been doing at COTAP and is almost at the verge of open sourcing it. I would also be sharing the repose. So let's start at first I would just like to provide an intro about myself. I work as a principal engineer at COTAP Data Engineering. I have around nine years of industry experience on distributed systems and big data. I started working with Scala since I think the end of 2019 when I was actually playing around with the Apache Spark which with the Scala API of the Apache Spark. So I started out and then slowly and steadily moved into functional programming. So apart from that when I'm not working you would generally there are a few things that I do. One is coding. I try to try out new recipes especially during the lockdown period. Really got into it. Also take an interest in playing table tennis and off late I have taken an active interest in chess. So playing on chess.com and Liches. Things like that. That's enough about me. Let's move on to today's topic. Okay. Yes. So let's talk about first about the motivation. How we came about creating these projects. So at COTAP where I work we have multiple heterogeneous data pipelines. When I say data pipelines these are data processing pipelines or data processing jobs which are run on Spark data flow which is Apache Beam, BigQuery, all sorts of pipelines are there. And then again these are SQL based pipelines and they also have custom processing logic. And when I say custom processing logic they're both written in Java and Stala. So that's where we started out. And the problem statement is basically can be divided into these two types where first is the coupling of the business logic and the runtime engine. So basically what I want to mean here is say suppose a job that is written in Spark cannot be run on Beam. Why do we need it? In today's world we are bound to use our cloud providers and these cloud providers hit a lot of bottlenecks like maybe cluster scaling issues or we don't get unavailability of on-demand nodes or simply we want to try out some other engine to see how the cost optimization or the cost figures are. But we are even if these jobs are written just in plain SQL, in Spark SQL, we cannot take that job and run it on Beam just because of the amount of boiler plate that is involved in it. And also the amount of developer effort that is required to do this coding. So that is something that was one of our thing, one of the problems that we wanted to solve. Second thing I would like to talk about is there was no code and functionality reuse. So what I mean by that is two things, inter and intra. Inter is between Spark and Beam. So basically say suppose I have written a functionality in Spark, I cannot take that functionality and run it on Beam. And also we also wanted to use the basic, the compositionality function, the compositionality of functional programming. Basically say suppose I have written some code which is in Spark, just do some good industry practices, put it inside a library so that it can be reused in some other repo. So within Spark code also there was a lot of repeated code, similar logic, lying in different silo which were all doing the same work. So first was simply have a well-defined library where things can be reused and then extended to something which is across platforms. That was the main problem that we were trying to solve. So let's see what was overall design consideration here. So one thing that you can see now is we want to standardize all the actions in the system. So when I talk about standardizing all the actions, we definitely need a domain-specific language, a DSL, in terms of which we can talk and then actions can be taken. And the motive of doing this is to do segregation of the platform and the domain. So the domain is the business language that we speak to do standard actions and it should not be tied to any platform, Spark or Beam or whatever. So we wanted that. And another very distinct feature that we figured out is within this DSL, within the AST, within the abstract syntax tree of my DSL, the atomic units, we needed some sort of combination. We needed sequential control flow. They needed to be combined in certain order where the output of the previous DSL is the input to the current DSL unit. So we needed that kind of a functionality. As you will see, the data IU and the ZO flow project has that. And again, we also needed DSLs which would run parallely and independently. The data quality or the data expectation supports that. So we are able to run independent computations, get all their results independently and then calculate something out of it. So this was, we were mainly looking for something like this. This was our design consideration. Okay. So what was our solution? So our solution were three distinct projects, three items, ZO flow, data expectation and data IU. So ZO flow is what we are calling the general model of computation that we have gone on to create. It reads data, it supports all kinds of transformations that we want to do on data. It also supports data quality and then you might want to send some alerts and you want to write your data. This is the general DSL that we wanted to create. Out of this, we figured out that the data quality in itself needs a separate DSL, needs a business language of its own. So we created another project which is data expectation which mainly addresses the data quality part. And then we had data IU. We figured out that the storage, you know, the blob storage, there's three on the GCS, read and writes are not that simple. So I'll give you one example. We have, you know, something called a zeotab specific feature. Okay. So when I say this, say suppose we have some optional columns. So when I read a data source, say I'm talking about Spark, when I read a data source, it was part data frame. I append some optional columns. Okay. If they're already not present in the data frame, append those columns and move forward because they might be needed in the downstream pipeline. So all these specific, you know, use cases, zeotab specific use cases or organization specific use cases is something that we wanted to support. So we created a separate business language for data IU itself. So having said that, so how, so now I would want to show how we used the functional programming constructs and how we use that to solve our problem. So first I would like to talk a little bit about the DSL part of it. So DSL as we know is a domain specific language, business language generally talks about what and forgets about the how part of it. Okay. It's not really bothered with the implementation when we talk in terms of DSL. We are just talking about how, you know, what is to be done. Okay. So, sorry. So our design consideration for the DSL itself was very, you know, clearly defined. We needed something that can accommodate a language with an AST like structure where I can define my atomic units of my DSL.  We needed to combine them in different ways so that we can get our desired result. And the runtime, the distributed compute which we are talking about or the execution engine would become plug and play operators. I would talk about this business language and I would plug in the spark or the beam interpreters at runtime and I'm able to generate the output. So these were one of our key considerations that we looked at. So having said this, let's move on to some of the functional programming concepts which we used, which we explored to solve this problem. Okay. So the first thing that I would like to talk about is reification. So what is reification? So reification is something which was abstract in your code. You make it concrete. Something which was implicit in your code. You make it concrete. So you take some concept that was implicit in the code and you turn it into data. You then can use it to be manipulated by your program. So that is basically what is reification. Let me show a few examples. Let me take the example of functional programming itself. So in object-related programming, functions are not data. However, in functional programming, you can pass around functions like data. So these were implicit. But now they have taken you can pass around functions like data and then you can manipulate that. Another example that I would like to give is the example of the option monad. So option monad really refies the concept of partiality. Earlier we would either return a null or a result from your code. But now we have defined them into a specific data structure which is your sum and none. And now you can use these data structures further in your code to manipulate to write a program basically. So that's what we are really looking for. So it is definitely it is everywhere. Clisly is another example of the reification of the concept of effectful composition. Trampolin is there in cats. It's reification of stack safety. And finally DSL which we really want to implement. So reifying the steps of a combination of procedure in order to represent them without actually executing them is a very common implementation strategy in functional programming. So we want to do that with our DSL. We want to take the atomic units represent them but not yet implemented. We want to defer the implementation to a later point in time. So let's see how we can do that. So first what I'm going to do is I'm going to show how we can go ahead and do something like that. So if you see here, if you had to write these functions this would be functions in your code. You load the source, you take some source paths and you return a data frame.  So these would be implicit in your code. We took this, we created some data structures out of it. We created an ADD out of it. So we have these case classes load sources and write to sync and they extend this flow DSL. So let's look at what our DSL would look like. So we have these are the basic atomic units of our DSL. Load the source, load the user defined functions run the transformations assert data quality, write to sync send alerts. These are the basic units. Okay, let's just skip to one slide. So now if you see what we did is we defined a trade flow DSL and all my ADDs, all my atomic units are now extending this flow DSL. So load sources, it extends a flow DSL and one thing to understand here is it returns a generic type A. Okay, because at a separate point in time, while loading the user defined functions, I might want to return a unit, but while loading, while doing the, asserting the column expectations, I might want to return some expectation result. So that's why we have kept it a generic type. Okay, so now what are we going to do? We have created this DSL. What are we going to do with this DSL? So I'll just outline the steps that we are going to perform. First is we are creating the DSL. We'll lift it into something called free. We'll come to that in the next slides. Then we'll construct the program using the DSL and we'll provide an interpreter for that DSL and we'll invoke the program in this structure. We'll invoke the fold map method on the program passing the interpreter separately. So these are the sequence of steps that we intend to perform. So before I move on, I think it's important to understand what a free really is conceptually. So let me move on. So one thing is if you look at the free structure, now if you see on the left hand side, you have this free flow DSL comma A. It is of the structure free S comma A. You can think of this free as the program which uses S, which in this case is flow DSL, your language to compute the value of A. So free is the program which takes your language as an input and computes the value of A. Another two more distinct things that I want to talk about free is which are there on the right hand side. So free kind of provides a monad for your AST or DSL. Now what does that mean? See the monad helps you kind of take the steps of the atomic units of your program and not provide an implementation of it to it. What you do is you take those individual why do you create these individual steps in these atomic units so that you can sequence them and you create a bigger abstract program within sequence and you can create a bigger abstract program. Now your AST is not really a monad. Now in order to sequence them you need something which is a monad. So that's where free comes into picture. You cannot use your for comprehension on top of your AST. So you lift your AST into a free which we can now understand definitely has a flat map inside it which is helping you to use it inside a for comprehension. So that's what we mean when we are saying that we're lifting your AST into a free because it will provide us with a flat map it will help us use it in a for comprehension. And another very important aspect that we need to understand it is separates your DSL from the interpreter. So you have written a DSL of which you have not given any interpretation any implementation as of now. These are just algebraic data types. You will provide the implementation in a separate structure which we call as the interpreter and these interpreters can be different at the runtime. It can be a spark interpreter, it can be a beam interpreter. So you can just call them. You can just call your program passing different different interpreters here. Passing different interpreters here. Sorry. Yeah. You can call your program using different kinds of interpreter here. That's what we are looking for. Okay. So having said this let's complete the usage of free how we have used it in our program. Let's move to that. Okay. So first I spoke about so you had your AST here. This is your AST flow process. You have to somehow lift it into a free right because you need the flat map which is free is providing in order to be able to write bigger programs out of it. So we have written something called free flow DSL which is nothing but a structure which holds your flow DSL and returns a type A. Now if you see here this is being provided by the free data structure the cats free data structure used. It lifts your AST what you have written the atomic units and load sources the right to sink into a free. Now one thing to note here is the output or the return type we have specified as free flow DSL which is this guy and not because if we don't do that with the output type would show up as you know a load sources free of load sources but while we what we actually need is a free of load DSL we'll come to that why we need this. Okay. So once we have lifted this now what we can do is we can so we lifted this because we needed the flat map why we needed the flat map because now we can start using it in a for comprehension so we can have multiple variants of our business flow one can be a very simple business flow where we say load the source run the transformations right to the sink a very simple flow another can be you can load the source run the transformations and assert data quality so I'm not really interested in writing to some sink I want to get some report on data quality and maybe my downstream pipeline uses it another one can be you know you assert the data quality if your quality passes then right to sink or send alerts so there can be multiple flows that you can create let me show you some example okay so there's a you can see a lot of code here don't focus on the code just focus on the highlighted areas what ideally want you to look into so we have two methods here end to end flow e2e end to end flow maybe we could have better name for them end to end flow and end to end flow with expectations so if you see here what we're really doing is we have taken our load sources load user defined functions run transformations right to sink and we have put them in a for comprehension so definitely these we have lifted into a free and somewhere we are getting a flat map where we are able to use them in a for comprehension and now we're able to you know combine them in multiple ways so that we can achieve the desired output so in this case that if you see the desired output is a unit after writing to sink there's nothing we can return so we return the unit but in the other case because we the part of this free flow DSL you know we had kept it a generic type a we are now able to return at times we are returning a unit here and at times we are returning an expectation result so that is something we need to know because they can be combined in multiple ways and we have kept it as a generic type we're able to return whatever we need at runtime okay at least that is how we can write the abstract programs so once we have done this you know till now if you can see we have not given any we have not spoken about any implementation we just wrote some see here we just wrote some ASD we took that ASD lifted it into a free now we wrote some programs some high level program and we haven't we haven't said yet whether we want to do this work using spark using beam using big query nothing we have not spoken about the implementation at all we're just talking about in terms of some business language okay so now we will be providing the implementation so if you see here this is what the implementation would look like okay it's a function which if you see it's a function K of it takes a flow DSL and returns an ID of K okay what is this this is called a natural transformation so it's really it's a flow DSL is a representation of your ASD you can think of flow DSL as the source ASD which is being transformed which is being mapped to your implementation to your target ASD okay or rather I shouldn't put it as ASD to your target structure in this case ID is the target structure right so this is where you do this is where you define your implementation this is a very simple definition we just printed it out we just took the load sources what we are saying at runtime just print out the source paths and when you at while you've taken you know when you're writing to things just print out that we are writing to see but this can very well be some other implementation which we can pass some other interpreter which we can pass which will actually do some computation maybe use it to write using spark or using the right so that's what we are trying to do so you know having so one thing to note here is having really separated the abstract syntax tree having really separated out the flow DSL here helps us to interpret it in different different ways at runtime so now I've provided the ID implementation I might as well provide which we will see as well later I might as well provide some other implementation okay so yeah and this this sign here is actually a standard sign that we use for natural transformation okay so I'll come to that later and how we invoke it we've already spoken about it you take the end to end flow you call the flat map fold map function on it and you pass your interpreter inside the general form is program fold map interpreter that's it you're done okay so since we have spoken so much about free you know we ought to see how does a free look like or what is the base intuition behind the free okay so we have we have spoken about you know lifting something into a free and then writing some programs using it and then passing an interpreter so we ought to see what a free you know looks like so first of all one thing that we want to understand here is we wrote our DSL which was not a monad right instead we gave all the monadic responsibilities to free right so free must be must be some data structure which holds the steps of our competitions so that later we can traverse it and we can interpret it at our build right so as you can understand like our flow DSL had an algebra right free must also have an algebra which it uses to store our AST right so free has an AST of its own right but from our previous slides we can infer that free must also have some monadic qualities right it is allowing us to use it enough for a comprehension so free must also have a point and flat map but one thing that we are what we are trying to really say about free is free is itself reifying the concept of a monad you take something and you lift it into a free you suddenly get to call flat map on top of it what is this this is the same concept that we saw before reification of the concept of the monad itself so why not how can you reify a monad so why not take what is there implicit in a monad like a point and a flat map and try to reify that right take what is there implicit in a monad the point and the flat map the pure and the flat map whatever you might call and try to reify that okay so look at this now let me you know see what is written here you know our AST of free should resemble a point and a flat map we are trying to reify the concept of a monad so somewhere it should definitely resemble this point and flat map if you see here we have defined this trait or this abstract class which is the free which takes a higher kind of type and then look at this pure there is any similarity between pure and point okay takes the A and returns you are free of FA right so this is one part and the second part is although the suspend part is not actually you know exactly looking like a flat map but the first part of it is right it stores are F of free of FA right which is this F of a part right so if you can see from here free would definitely be you know a recursive structure where it would look something like this okay where you would have this recursive structure of suspend where F would be your AST that would be stored inside the suspend and it would be terminated by a pure right so suspend really what it does it uses it applies the F on the previous suspend to calculate the free okay so it's definitely what we can see is it's the terminal point here is pure and it's a recursive structure and A is basically the recursion carrier here so this F of A this A is basically the recursion carrier here okay so okay having said this let's see so once we have come to this definitely free is a monad so free should also implement the interface of a monad the point and flat map okay so let's see how it implements that okay so as they say in functional programming when you don't know what to do it's best to follow the type okay so first the point is really easy to implement if you see here we have this see great free and the point is really we can just take the data structure pure which we defined earlier here and we can just wrap the A within that pure right flat map will be a bit more involved let's look into that so first is we'll just follow the type so first we get the pure right so what we're going to do is we're just going to call we're going to pull out the A from pure and pass it to the F that's it you're done with pure okay next we have to implement suspend so since this returns a free it only makes sense to wrap it in a suspend whatever code we write it will be wrapped in a suspend because we have to ultimately we have to return a free right so next let's see what are the operations that are available to us that we can apply here so one thing that we can definitely see is we have a free available over here and if we have a free we can apply a flat map on top of it and pass the F inside so what I mean is this is the structure so the S is F of free of F right so if we can somehow pull out this free from this F of free we can always call a flat map which is this flat map itself we can always call a flat map on top of it and pass an F okay now the question arises how to pull out this free from this F okay so the ubiquitous function that we use to pull out something is the map operation right so we can use a map on top of S right we can use a map F dot map or S dot map and we can pull out the free with which we can call the flat map but the point is S is not a functor right we are not getting any you just this S is basically if you really look into it if you really look into it this S is really your F which you have defined which is your AST it is not a functor right so how do we do this so what that means is implicitly this is the only requirement that we have for a free implicitly we need a functor on your F a functor on sorry a functor on the AST that you have defined that's the only requirement so question is we have not provided a functor on F how would we do it okay so cats does that for you it provides you a functor on F okay so in principle if you see free is basically something which takes a functor adds a pointed part to it adds the monadic behavior to it the flat map part to it right and this functor itself is provided by cats okay there's a neat trick there it's really a lemma which is called Coyonada I'll not be covering that here but that's probably the last piece which you need to know this part is already implemented in cats in a structure called flat map and that's why you get a functor out of it so what is a Coyonada at a high level it is basically if you provide it a higher kind of type constructor which is this this is your higher kind of type constructor if you provide it that it will provide you a functor for free okay so that's how you get your functor and you are able to use it in your flat map as simple as that so if you guys are interested please go ahead and take a look at the Coyonada lemma and yeah that's been provided by cats okay alright so we have covered a lot of theory one second so now let me reiterate what I have said in a little more technical term so what we have done really is we have defined a higher kind of type right your the AST which we defined is really a higher kind of type F of something okay we lifted the higher kind of type to a free we wrote an abstract program using the DSM till now there has been no interpretation next we defined a natural transformation which was really F of A to G of A okay so what is this natural transformation it is really a function which you write from F of A to G of A what does this really mean the F of A is really a source a blueprint of what you want to do the AST which you have written is really a blueprint a model right representing something and the G of A is the target interpretation you are targeting it which you have written so if you see keeping this A constant maintaining the structure of A it is helping you to convert one higher kind of type to another higher kind of type so it is called a higher kind of type of first order higher kind of type what that means is here if you see this is a type constructed with one whole when we say a list it means it is a type constructor of one whole so I am giving just an example here when I say it is a list it is just a type constructor with one whole it is not yet a type when you say list of it then it becomes a type so what natural transformation helps you do here is it helps you convert a first order higher kind of type F of A to a G of A so here you have a type constructor with one whole okay it helps you keeping the structure of that A constant it helps you convert it into another container it helps it convert it into another monad or another structure that is what natural transformation really provides for you so if you see here same example there is some data expectation which we will look at next which we have defined is converting it into a Spark data evaluated both of these how do you know it is a higher kind of type it is a type constructor with one whole okay that is what we are saying so let us quickly look at the next set of items which are free applicative because as I said there is one thing that I mentioned at the beginning is although we needed for comprehension for sequential operation we also needed operations which were independent in nature okay so I will give you an example of these independent operations see suppose in our data set we have a column called gender and the gender column can maybe there are two things that we want to know about it one is is it non-null or not it should always be non-null another thing it should be one of the values between male and female okay now whether it is non-null or not has no implication on whether the values are male and female or not we want to compute both of them independently and then combine their result we don't want to do like once we have asserted non-null then only go ahead and compute whether they are within this value okay so these kind of independent computation is something that we were also looking to represent in the data quality piece and hence we wrote this interface which is our data expectation more or less it is the same interface with just a small change of how we are combining so I'll show that so here you have data expectation we have written some constructs on top of it should be present one of and these are the entries maybe male female and non-null always okay next what we do is if you see it is exactly same instead of defining a type with free we define the type with free applicative okay we put the data expectation in place of s and then we have a so this is the type we wanted to define validation and then free applicative also provides a lift which actually lifts your ASD lifts your DSM into a free applicative so it's exactly the same so you just lift it once you've done that now is the only change that you'll have to keep in mind so we want to check whether column is present or not and we want to check should have one of male and female and we also want to check should always be non-null okay these are examples of two programs so we use a structure here called as map N okay instead of for comprehension we use something called map N what this means is this is really a product where all of these will be calculated independently and the result of them will be joined in some business logic that we provide so here we wanted to add that maybe this is a Boolean and we are saying that only when column is present and it should have one of these entry and it should have it should be non-null return a true something like that okay so this is the only difference that we have okay and this is the implementation again exactly the same one thing to note here is the the the hierarchy type that we are using here spark data evaluated okay this takes a data frame and returns any so a data evaluated as we will see a little later that it can be of multiple types the interpreter the output type of the interpreter can be of multiple types so this is one thing that I wanted to point out here okay now invocation is exactly same program full map interpreter so you have your mandatory column checks you do a full map and you pass in your interpreter here okay I am sorry only thing to point out the applicative and free monad is free monad you will use when you have dependent computations and free applicative you will use when you have independent computations like we had for data expectation okay let me show you an example of the interpreter and then we will probably wrap it up so alright so before we have written a lot of interpreters in our project and we have made heavy use of the state monad so let me just briefly explain the state monad first which is in functional programming state monad is used to handle application state so what I mean by that is you start with the state and you calculate so you start with s and you calculate s comma a where s is the updated state and a is the value that you calculate okay the result that your function produces so it's not really s I like to represent it as s dash okay where s dash is the updated state okay so when you put it into a for comprehension the previous state which you calculate is passed in the next computation to be used okay so that's how the state monad at a high level works so I'll just show one example here so if you see here we are using a state monad for you know so what we are really doing is we have taken some data frames we are loading the data frames then we are running some transformations on them then we are running we are loading some user defined functions and when the sequel is running they are using those into the defined functions then we are running some computation if you see here this is how we use the state monad to do these computations we have the state monad here remember the the thing that we have to keep in mind is s to s dash comma a which is you started with a state which is this context okay so let's say we are calling the for comprehension in this order load sources load user defined functions run transformations assert column expectation right to sync let's say they are in this order okay so when you start with a state first I will pass in an empty map let's see first I will pass in an empty map if you see here an empty map is being passed we loaded the source okay so this s here converted into it was an empty map it converted into a map of string and data frame here you actually have the loaded sources with you and in the a I just passed back the empty map okay which is really this line here so the context if you see the context is passed as a a which really does not have any impact but whatever state we loaded right whatever sources we loaded we passed it in this left-hand side in the s dash part okay so next if you see next we have the user defined functions so generally in spark we have the user defined functions return a unit it does not have any change or any bearing in the current state which is your map right so nothing changed in the map whatever map came here we just passed it as it is and we loaded some user defined functions and that's why we have returned a unit here so whatever we are calculating we are kind of returning that so we have returned a unit here okay next step let's see next step we have run transformations okay so we are now we need whatever map was here whatever map was passed here we ran some transformation on top of it some queries which will transform it in different ways so if you see here we passed this map and the enhanced map was created this will have more data frames and you know more data frame names inside it and on the right side we just passed whatever came from the input okay once we did that the enhanced map is now the input enhanced map is now the input to the hazard column expectation part so here we really don't do any transformation or we really don't change the state what we instead do is we calculate some data quality checks on top of it so we run all sorts of data quality checks as part of data expectation which we have shown we run all those checks now if you see in we are returning a column expectation from it okay so if you see here we have returned we have run the column expectation and the output of this guy is actually a column expectation which we can choose to use or throw away whatever right but the initial state remains unchanged and finally we pass it here where we are writing to the sync here again the state does not change but we write the output to you know some external storage okay some blob storage and that's what we are doing here we are taking the map and we are writing it out to some external storage so that's about it okay that's how we are using the state monad okay so one point one note that I would like to make here is when you are writing these interpreters the most popular choice we saw is the state monad because what you are really trying to do is why have you written this for comprehension you are really trying to pass some previous state to the next computation you are trying to take the output of the previous atomic units atomic unit and pass it to the next computation and that's why in most of the cases state monad is the most popular choice probably but then we also used a lot of these readers okay the reader monad so if you see here we have reader of data frame writer this is the spark data frame writer reader of data frame reader and then we have reader of p collection this comes from beam these two come from spark we have used a reader monad here what a reader monad really does is it takes an environment and calculates something using that environment okay but in this case also this data frame writer and this p collection reader really stores a state inside it and that's why what we really did is we took the data frame writer set some state on state on top of it okay so the data frame say reader for spark you want to mention whether you want to load a csv or not so you just set an option csv that's it so the entire state thingy is maintained by this data frame reader so we did not need to use a state monad but otherwise here also popular choice our popular choice would work you know we would be using a state monad just because these guys are storing their own states we use the reader and we make made them you know independent so that's one thing but otherwise also one point I would like to say is we have to be creative while you know writing interpreters it can even be a function like I was trying to show earlier in case of data expectation what we really wanted to achieve is we will have different different you know interpreters where where the type itself would take a data frame and return a if you see this is the type this is the type with one whole where a is something which is really malleable we can change that according to our wish and it itself returns a function which takes the data frame and returns a function okay so you can even be a function your higher kind of type need not be a data structure it can even be a function so you have to be creative there whatever fits the bit basically so yeah that's about it I think I'm done with the time as well so these are the repositories that we have open sourced till now so zio flow and data IO has been marked public so the first two data expectation is not yet public we will be doing that very soon maybe in the next coming weeks for some people who are using spark we have also open sourced spark property tests library which is to write better property tests on spark and it is available on maven as well so please go check it out and we have Gitter channels and it threads as well so you can comment there and tell us about your feedback so yeah that's about it and yes thank you for joining this talk thanks for sharing your experience with us today


Hello everyone, I'm a staff from permita. Yo, and I'm going to talk about opal the open policy administration layer And now it can help build Cloud native permissions more easily Sorry about that So you just got your project to build in-house permissions But you need to understand permissions are hard okay, so There's a lot of modern challenges like micro services Infrastructure was not what it used to be 10 years ago And now I cannot just embed permission logic into my application because I need to enforce access across many services Some of them are third-party services as well Modeling permissions is really difficult while all developers almost know what role-based access control is Eventually you'd want to move to more complex systems. Let's say you want to get access based on the user location You cannot do that with or based access control You probably want to move to actually with based access control or if you want to get access based on ownership More fine-grained permissions. You probably need relationship based access control. So this kind of challenges are Are difficult because these systems are difficult to grasp and are difficult to implement and Finally, there are mounting security and compliance standards that we need to adhere to we all like our privacy But it comes with a price and implementation. We need built-in auditing. We need checks and balances for example I want to be able to say who can even change the policy Inside my application. So of C for of C. This is also important And all of these challenges are the reason that broken access control is the top Top issue in OS top 10. So 94% percent of applications were tested for broken access control And we want to avoid these pitfalls in our implementation. So the first one that we see Most I'm sorry most commonly in coupling application logic and authorization logic You may want to start with just like admin and not admin for permissions and that's okay But if you encode this logic in an if branch in your code, eventually you'll get this code block And while this is a like completely fine for monolith applications when you get to micro services it completely breaks because You'd need to duplicate that logic across all your services and duplication leads to drift and drift leads to Security incidents. So we don't want that. We want to put everything in one micro service in one place The second kind of thing that we see is mixing up authentication and authorization. So We see implementations where people just take Claims from octa and just put them in json web tokens and just check that in code And that is fine, but it has a few downsides So organization roles are not the same as application roles if I'm the head of IT I shouldn't have admin access to the financing app. So some translation from organization roles to application roles needs to be made and Another thing is that Jots tokens are not storage. They are limited by the Http header size. So eventually when you'll have more fine-grained access you will exceed this limit and lastly This is not flexible if you want to change the permissions for a user mid-session because the user has a job It logs in everything's fine next time you re-logs in this will be applied, but this is not flexible enough and Finally Not planning ahead. So this is completely fine like starting with something simple But let's hope you do better and the app will grow and more demands will come. So demands from customers You'll have This really great deal that could be great for the company with an enterprise customer And they need finer-grained permissions and now you need to build that into your application really fast or You want to expose API keys through your application. You need API key management All of these demands Will come and you want to build a system flexible enough so that it can grow over time without being so Painful to rewrite from scratch every time So Let's talk about the best practices really briefly you want to decouple policy from code You want to be able to respond to events real-time? Okay, you want to have the github's workflow for policy You want back office for stakeholders and you want interfaces for customers and I will go I will go over them one by one. So First Decoupling policy from code. We already agreed. This is a great idea And really cool project that most people know right now is OPA this is the most adopted policy agent currently in the market and OPA lets you express policy as rego code and Data is JSON and this is great. You can just create all the policy in the organization in rego and just manage that OPA does have one downside. It was Originally made for Kubernetes admission control and it cannot actually respond To events in real-time. So for applications, this is breaking and that is why we made OPA So OPA is an administration layer for OPA that gives you two main things First github send second real-time updates. So I will go over them with OPA Each OPA client manages a single OPA agent and then it can be managed by an OPA server That can run in a cluster. So the OPA server's job is to track a githrepo and Serve that policy from there It also can push real-time updates like JSON patches to the OPA client and from that to the policy agent and Finally OPA client can actually access data that the OPA server cannot you can send any Instruction to OPA client even if they're in a separate network to bring data from let's say an in-house database That shouldn't be exposed to an outside network OPA is extensible it has a plug-in mechanism called fetchers. So we already have fetches for LDAP MongoDB Stripe etc and you can use them if you want everything is open source So OPA gives you the ability to manage policy in gith. Why is this so good? Yes, so With githops for policy your policy is auditable You can see who made the changes where the authors you can see all the you can use signed commits and entire history is available to you It's immutable rolling back is easy. It's super easy and also it gives you the power of CICD you can have tests you can have approvals with PRs for something sensitive you can have multiple approvals and You can have quality signals So with OPA you can have policies that are not as performant as other policies There is a meaning how you write your policy So you can just have a code signal in your CICD that says if the policy is too slow It will affect my app SLA and please let me know So all that power is available to you if you manage all your policy agents with OPA So if you want to talk more about off Z Please come find us at boosts you 25. We're also available for slack and Opal is available at opal.ac. It's Apache 2 So, thank you very much