WelENCE PLEGI Hello everyone and welcome to what I suppose really is a special business whispering because today I'm joined by Paul Greg for those of you are quite young. You might not know that Paul was for 6 years, a board member of Everton Football Club. Paul has a background, that very impressive background actually in business. He founded, you had a family business, what might call it, which he grew to ein lleiadau mwyaf i'r cyflwyno ymlaen o'r ddigonion yn y Caerdydd Unigodd. Mae'n Ie, mae'n rydyn ni'n meddwl i'r bwrdd yn ymgrifennu'r awr, i'w ddweud yr awr, ym Mhwyl Gwll Ddechrau, a gydag o'r stwyff honno yn ymgyrch yn dweud.  So gaeli'r pawr yn gweithio i fy nghymru. Rwy'n ei gweithio'n gweithio. Rwy'n dechreuwch i'n gweithio, rydyn ni. A dyma chi'n gweithio. Rhywodraeth ymlaen i'n gweithio. Rhywodraeth. Felly mae'n meddwl, mae'r city llwyddiad ym gweithio. Mae'r Llancastrwy a'r Bydd, felly mae'n meddwl i'r gweithio. Felly, rwy'n meddwl, nid yw'r hynny, mae'n amser i chi wedi'i meddwl i Eftan, yn y ddau ymgyrch yn ymgyrchol, mae'r ddweud i'r Ynw'r Yorkshire. Mae'n meddwl i'r llwyddiad? Mae'n meddwl i'r Llancastrwy, mae'n meddwl i'r Ynw'r Yorkshire. Yn ddwylo'r ffordd, ac oherwydd, tynt o'r ddweud yn supporters, dyfed nhw lleoedd o'r llyfr. Mae'r llyfr wedi'u mwyaf arall. Mae'n rwyf. Ond ydybodaeth eich gweithdoch gyda'r hwn. Mae'n dangos, mae'n lleffaf i'r gweithdoch. Well, I alluded to the fact that you're a friend of Bill Cernarad, because Bill was, like yourself, would put shows on and obviously needed theatres and those sorts of things. How indeed did you get involved with what ultimately was True Blue Holdings, which was obviously the vehicle for buying the football club? Yeah, I think that was one of my mistakes. I didn't quite understand True Blue Holdings and I thought it's what Bill wanted to do and the other directors. But what that did was it frustrated your work as a shareholder, because effectively it put Bill in charge of the club. And I think the thing is that as we progress and I've been independent of True Blue, then we could have had a great impact on the club. That's interesting and that leads naturally to when you did join the board of the football club and they were obviously quite interesting times about eight years or so. We're talking about year 2000, aren't we? Around eight years after the formation of the Premier League, football clubs like Arsenal were still playing at, you know, a hybrid and things like this. What were your aspirations? What did you think you could bring to the party and what could you see in the near to medium future for the football club if it was able to frankly exploit your experience? I think that when I joined, or when Bill asked me to get involved with him, I took a view that Everton was an opportunity. I didn't quite understand it at that point, but I thought, well, if it's good enough for Bill, it's good enough for me. And so we put our money in and we got on with it. But I think once I was there, I realised that this was a sleeping giant tucked away. And that one of the challenges was to find a new stadium because, goodness and par, as much as it's loved by every Evertonian and every panel goes there, it's traditional hope to walk around. And that's the trouble, everything's been an old traditional club for too long. So you said that had you been independent, what was the structure around the holding company? I mean, was it equal voices? Did you feel like you didn't have enough influence? What went wrong almost? Well, I think the thing was that Bill invited two of the colleagues to join the board and they put some money in at the same time as I did. And he made one of them a deputy chairman and one of them was the director. So I think in a way they were the decision makers. And I think that in hindsight, you couldn't have a true voice. I think the thing was that part of me wanted to see what we could do with the stadium. And at the time that time, Liverpool had just been given half a million or a billion from the government to redevelop the city and bring it up into speed. And we went to the council and said, what about a new stadium? Multi-purpose entertainment and sport, and they supported it. And we spent nearly three years working on that with Liverpool vision with the council to get the King's Dot to a point where they said, yes, they would do it. And then it all got blown away. So it was a real frustration. And then maybe if True Blue hadn't existed at that time, we could have had a shareholders meeting where people could have been independent of True Blue. OK. And so that vehicle sounds like it became a constraint and said that you've written your book and hopefully people will buy it. Perhaps many Evertonians will simply because of those five chapters. What do the five chapters cover? Because if there's five different chapters on our great football club, I guess the five things that you think were most relevant to your time on the board? I think the main thing was the King's Dot opportunity. Obviously, yes. They usually missed opportunity. And despite all my pleading mobile, he said, I have to own the stadium. I said, but you're leasing a stadium for 125 years. I said, you've got no capital risk. I said, your lover spending all the money on the playing field is not frustrated because you're going to a new stadium. But I said it will increase the opportunity. And the stadium was designed by the people who designed the Spurs Stadium. So we were like stayed to the art 20 years ago. Well, what struck me and I'm a bit younger than you and a hell of a lot younger than the rest of the people in this studio. And what struck me back in the day and if we talk about 2000, I would have been in my late thirties or something was I personally in those days. By business trader, IT technologist and ultimately ended up being a big player and outsourcing those sorts of things. I didn't have that emotional need to own the stadium. And of course clubs like Manchester City at the moment and West Ham don't own their stadiums either. Was built being just crazily short sighted stuck in his ways traditionalist and was this really a stumbling block between someone trying to look into the future and football fundamentally being an entertainment industry and having a multi-purpose stadium against a guy thinks no, it's our club and we have to own the ground as well. Well, I think, I mean, I used to say to Bill, I said, look, I said this stadium has been designed that we can put Madonna on tonight and we can play Arsenal tomorrow afternoon. I said, imagine all the stars when they do their uncle going on stage wearing Everton scarf and hat. The publicity the brand will get will be enormous. I said it will outshine every other club in the Premier League, but he didn't get it. And I think that in hindsight, I think he just loved the old Goodison Park. He used to arrive in the boardroom, but the men on one side is why we take the ladies on another side. We have the traditional Everton cake and it was just like, that's what I think Bill really felt comfortable with. And maybe he was scared stiff of a new stadium. I don't know. It was the biggest failing in our history, probably, even accounting for the current difficulties we have around financial fair play and those sorts of things. And the debate goes on now, of course, with the potential change of ownership about will that mean the football club does or does not own the stadium? And many Everton fans perhaps would agree with what Bill was thinking 20 years ago. It just seems an eternity ago in my mind. But weren't there other people on the board who shared your views or were you just totally outnumbered? I think in hindsight I was totally outnumbered. I mean, we had a meeting a couple of days before the AGM and Bill brought in his advisor. We went through the secondary mortgage of £30m and he said to Bill, no, I think it's okay, you should do it. You're going to get a brand new stadium, you should do it. And Bill said, right, okay, we're fine. Then on the Thursday it ended the AGM and announced that he wasn't taking any secondary mortgage. He wasn't going with a new stadium and that was game setting match. And I think that I was gobsmacked at that AGM. That's what he delivered. I just couldn't believe it because he knew how much work had gone into it. He knew what the opportunity was. And I went to David Henderson the following day and said, look, Bill doesn't want to do it. He doesn't want to raise the £30m or take a second mortgage. And David Henderson said, look, we want this for Liverpool. We desperately want it for Merseyside. We want it for the year of culture. We will put the money in. And I agreed with him that we would give the council a 10% stay in the revenue for 125-year lease, which is what he agreed. And also, Bill didn't understand that we owned half the stadium with the city council anyhow. But he didn't get it. And he said, no, I'm not doing it. Was that not a board decision of the football club? No. So was this Bill going off-piste and just doing what he thought was right? Well, he may have spoken to Arthur Abercrombie and his deputy chairman and got their approval to him saying it. But it was, you know, it's like having been relegated. Now, would everybody feel about that? That's how I felt about the stadium. I couldn't feel any worse, you know, because I just thought it worked the effort. And also, what would do for the brand? I mean, the brand value thing, I think, is where the huge missed opportunity is. And I don't think you can overstate, as you say. I'm sure every single time, let's call it a pop concert, it takes place at a football stadium or indeed any sporting stadium, there's an employee somewhere, as you say, with a scarf in the right colour.   Probably doesn't know anything about it and he's waving it, you know. So it must have been massively frustrating. So what was it like for you, and indeed for Bill, I guess, because you must have been at loggerheads quite a lot by the sounds of it? Well, we were also going to loggerheads at a point when, you know, I suggested that he should stand down because I just felt that he might be a great Evertonian, but he was not a great chairman in my opinion. And it's sad to say that now he's not around, but you know, the point is that for 20 years, look where the club is today when from his chairmanship start and his ambition for the club to where it is now. It's not been a particularly good journey and of course, during that time, he may have lost your confidence 20 years ago, but he lost the confidence of many an Evertonian in the years that followed. So it's a shame, but didn't you get quite close, frankly, to displacing Bill from his chairmanship on the board? We tried. I mean, it was, and I'm not saying we could have resurrected the stage at that point, but there were serious frustrations in our position in the league at that time. I think we were like 17th or just on the edge of going down, you know, and I just said to Bill, I said, look, there's an opportunity to bring in a new manager, which I think will help to get sales and do that, but he just wouldn't have it. I mean, Bill often got accused by the wider fan base of thinking the football club was his and he could do what he wanted. It sounds sad and you're right, and I knew Bill reasonably well that he would class him as, you know, a proper Evertonian, but he himself in the latter days, when I used to speak to him, recognised the mistakes of the past, but the thought that a single individual could do stuff when you have a board is pretty damning and in some respects, part of the reason we're in the mess that we're in now is that the football club itself as an entity has allowed Farhad Mishiri to behave in a similar way. It's his train set, so to speak. Looking back, do you think there was any chance of you being able to get agreement or displace the people who were short-sighted if I don't put words in your mouth? Or was it just something in the moment and we lived with the consequences? Well, I think the other thing was, you know, there were offers to buy my shares, we got into the situation, but it didn't represent what I'd paid for, and then we found out that somebody else had been trying to sell them for a premium but not share that premium with me, and that created another raft of unhappiness, and I think, look, it was sad because I think the one thing I learned was that Everton is a club of Liverpool. Liverpool is a great club, but it's more international with its appeal and audience and everything else. I think Everton is the true heart of Liverpool. I think Everton is the true heart of Liverpool, but I just honestly feel that that point was missed, and I learned a lot. You know, when we were struggling in the league, we used to get letters of people. I remember one lady wrote and said, my old man's getting up at half past three in the morning, he can't sleep, he can't do his job, can he sort this, Evan, come out? And I think that was a genuine feeling among many of the Tonyards, that we, as a board, were delivering. It's funny you say that because what goes around comes around, and that's where Everton fans are right now. The anxiety levels go through the roof of the football club, and we haven't won a football match for three months. Felly, oherwydd mae'n edrychwch. Ond wrth gwrs, dweud i'n dweud o'r ffordd y tafn yn y cfflu. Mae'n ddod o'r ffordd yma sy'n sylfaen i'w ddechrau yn y Llywodraeth. Mae'n gwneud y Llywodraeth i ddweud oherwydd y bach oherwydd y bach neu ddechrau yn y Llywodraeth. Yn ymddangos llyfr yma, mae'r gweithio'n samur yn y Llywodraeth. Mae'n ddweud i'r llwylogaeth fydd yn cael ei gwaith o'r dweud o'r llwg?abyd am y hawdd? Cymru. Fawr AMMA, fod yn ddoch yn rhan a chymru i wych yn gwneud a gwnaeth, yn arun, a mae'n fyddio iawn o'r ysgol. Mae'n dweud... Mae'n dweud, a yna, yw'r ddape y sydd yn wedi bod yn y dyfodol. A yna yn ddape'r ddape, Metw'r Rhaglen yn digwydd. ac mae'n dweud ond yn gweithio'r gweithio a chael ei wneud yn ei fawr, ond yw'n gweithio'n gweithio, ond ond yw'n gweithio. Felly, mynd i'n credu y gallu amser yng Nghymru a'u gweithio'r newydd, yng nghymru, a fydd yn gweithio'n gweithio. Felly, mae'r gweithio'n gweithio'n gweithio'n gweithio ar gyfer â ni. Mae'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio. is probably prevalent today as well. But, while all this was going on, there was obviously people scrambling around for money, particularly to satisfy the needs of the preferred status. Was the money available, Paul, from you or from other sources that you were comfortable with, even though ultimately we failed to prove it? Rydyn ni'n olygu'r ffordd yma, y bydd Bill yn ddull i'ch ffordd o'r llwydd ac yn sgwm y plwyd. Rydyn ni'n olygu'r ffordd o'r cyflwyno'r rhai oedd yn ddweudio ac mae'n olygu'r hyn. Mae rhai o'r peth wedi'u ddweudio i'r Phefyd, ond rydyn ni'n olygu'r peth wedi'u ddweudu o'r peth. Felly rydyn ni'n olygu, ac rydyn ni'n olygu'r peth yn olygu'r peth, rydyn ni'n olygu'r peth yno'r peth. Felly, yn gallu gyda'n goll, ac yn gallu goll, byddai'n gwybod y byddai fath gwaith ymlaen nhw'n fawr? Rwy'n gwrth i'n vivu. Ie, maen nhw'n sgrif hon. Mae'n rhan o'n ffordd yn ei ddweud. Mae'r gwybod beth yw'r ysgol yn ar gwyfodol, ac mae'n gweithio'r sinful i'r bobl yn gweithio. Mae'n ddechrau'r llei, ond mae'n rhai gyrhau, a'r gweithiau am gyfaptor, I am gyda y tu allan, mae'n meddwl, dwi'n meddwl allan, a'n meddwl gyrsio'n meddwl. Mae'n meddwl i ddif triesio'n meddwl eich bod ni wneud yr un peth yn eich meddwl, ond, ond rwy'n meddwl honno yn 2004, yna ei gael o'r rhan o mechanism iddo. Yn y tro'n mynd i'r pryd? Mae'n meddwl ei'r rhan o ddamlu ar hyn o bwydfan deolch yn yr Hathor, Mae oeddo i dda i fod i'n f aidwaynuaeth. Doedd am gyd am eu gyd o'r ddweud y mynd yn eu gwir yw ein amser a ulyg â Samuelson. Yn cyfnod yng Nghymru. Mae'r gysylltu Samuelson. Cyfnodd dda i ddim ddechrau y Gymraeg a rwy'n cael ei ddwy. A gynnwch eich ddweud o phil. Gwyn o'r ddweud? Gwyn o'r ddweud? Eryddo yn cyd-dweud o gyd-dweud sy'n gyd-dweud? Ac mae'r ddweud o'r ddweud o'r ddweud? rydych yn bai gyd yn y bAN, roedden na fydd yn swydd, ac nid o holl o'n, dyna mae'n dechrau. Ond mae'n gychwyn cymryd, ond mae rhywbeth ei hyn. Yn fe, rwy'n rwy'n rhywbeth ei hyn. Pryddech chi'n dda ni weld, rhywbeth eich iddi'n mynd i'n rwy'n sylfaen, yn rhaid i ei wneud ychydig yw'r rhaglion yw nod yw'r rhaglion ar gyfer methu 777? I'r perprwethaf newydd ar y dechrau, ond mae'n meddwl gan effaith rhoi'r cwîdd felly'n gweithio ar y ddechrau'r fforddau newydd yma, ond ond byddai'r fforddau'r dyfodol ac yn ymgyrch ymgyrch o'r fforddau'r Llyfridog, y fforddau'r rhai cyfaint o'r ddegodd yng nghymru, ond mae'r fforddau'r defnyddio'r hyn yn teimlo i'r byd. A'r hyn o'n gweithio'r cyfaint i'r syniadau sy'n gyfrifio a'r cyfrifio'r cyfrifio a'r hyn o'r ddegodd goeth gweithio y ddull, ond ond rwy'n ffordd yn ddweud y cerddoedd, ond ei fod yn bwynt, ei fod yn bwynt. Roeddwn i'n dweud ymlaen i'r cyffredinol ond mae'n ymddangosion i'r gweithio'r gweithio'r modysgol i'r ffordd yng nghymru, a dyna ni i ddweud. Roeddwn ni'n ei chweithio ymwysig o'r weithio i'r gweithfyrdd John? Roeddwn i'n gweld i'r rhan i'r piwp. Roeddwn i'n gweld i'n gweld i'r gwahol, Ac rwy'n mynd cael ei fod yn wneud i'r ddweud o bwysig i'w clwb. Mae'r ffordd yn fwy, yn ei fod yn ei wneud. Mae'r ffordd yn ei wneud i'w clwb. O'r ffordd yn ei ffordd, mae'n dod o'r cyd-fwrdd. Mae'r cyd-fwrdd wedi amfysgol y tro. Mae'n rhoi i'r ffordd yn y tro, mae efo'n ymddir i'r parw sydd. Dwy yw'r cyd-fwrdd? Fy enw? Dwy yw'r cyd-fwrdd? Fy enw! Mae'r broses erbyn yn ystod o'n gyfnod o'r quesig oedd o'r Cymru yng Nghymru a ddod o'r Cymru, fe fyddai'n hynny fyddai'n gweld ei wneud o'r cysylltu i'w cysylltu. A oedd o'n ddod o'r cysylltu i'r gwirio? O'n ddod o'n ddod o'r cyfnod o'r cyfnod o'r cyflwyno? Cynnyddu'r cyffredin? A oedd yn ymgrifio'r cysylltu i'w cyfrifio? you may not have started life as anewttonian, but you became one, did you retain that interest or was it just a business thing that happened for five or six years? It was a business thing there for five or six years, but when you're being involved you can't look into the results or sharing the results seeing where the team is going and you get worried when it's you know it's struggling. a'r dyn ni wedi bod yma'n rhaid i'ch wneud, pan ydych chi'n ddyn nhw'n gweithio gwerthu'r cyfathol, a'r cyfathol sy'n gwrthod cael eu bod yn hynod. Ond, yn ystod, rwy'n gwneud mewn rhaid, ac rwy'n gwneud yn gweithio'r rhaglenio i'r dweud cyfrifiadau. Yn ystod yn ystod yn ystod, Aberton, ond y gwaith yw'r rhaglenio i'r mwyaf ar gyfer ymgyfrifio. Ond rwy'n gwneud yn gweithio'r rhaglenio i'r ddysgu. it was a great loss really and we can only punish ourselves by thinking about what might have been because clearly we would have been one of the first movers in you know, a modern-stage in multi-purpose stage, a pitch goes outside on the banks of the Royal Blue Merci. Right by the city centre and all those sort of things.甁 ofimetr anhauciad all round the pitch They are 300 days of the year. Felly Evertoryn gadaeth y bryd yn ymgyrch. Eftan o dda, yn ddim yn fath. Eftan o ddechrau, dwi ddim yn dweud. 3, 4, 5 ysgol, mae'n gweithio bod Evertoryn Ffwrdd mae'n gweithio i fynd o'r ffrif, mae'n ffraith sydd wedi'u gweld a fyddio'r cyfnod oesod o'r cyfnod o'r cyfnod o'r cyfnod o'i ddau Brynlymor. Felly mae'r ddwy i'n ddefnyddio gwnaeth eich bod ymddangos o'r stads iddyn nhw? Felly, mae'n gwneud i'r ddael yn y stadium neu hynny'n gweithio ddweud, Cresol Paolus yn y Cymru, yn gallu bod yn ymddangos. Felly, mae'n gweithio'n gweithio'n gweithio yma? Felly mae'n gweithio'n gweithio. Felly, mae'n gweithio. Felly, mae'n gweithio'n gweithio. Mae y gallwn yn oed yn ddweud, mae'n gweithio'n gweithio ddweud â'r stadium, ac mae'n cyfnod o'i cyfnod o'r Gweithio yma o heta. Merthaeth ar ychydig yw'r enghreifft. yn gyntaf Gwyrdysigfark. Rwyf gofyn ni'n dweud o'r gael yma. Felly, ddim! Mi'n fwyaf, rwyf, dwi'n gwylliant, Paul. Rwyf gofyn, y gwerthgybiad. Rwyf i'n gweithio'r gwerthgybiad i'r gwrthiau mlygonu i llwyddiol. Rwyf i'n ddim yn unig derbyn gyda yng ngharifft yng nghymru yn Gyllidog ydyn ni'n gweithio i'w bwysig, byddwch yn gallu gweld ffordd o gyfod arno eich newydd. Ond y newydd yma yn ymlaen i ymdweithio ac y ddweud y ddweud yn ymlaen i ei ffodol, ac rwy'n mynd i ddweud y chyfodol iawn i gael eich ffodol. Ac rwy'n mynd i gael eich gweithio eich dweud yn y norweis. Rwy'n mynd i ddweud ymlaen i'r ffodol, rwy'n mynd i ddweud ymddangos ymgylch. Felly y gallwn ymdryg ar hyn o gwyfodol ei gael ychydig. ac mae bethувau angen. Fe wnaeth yr ydym yn ymgyrchu yn LL, fel yna. Beth ydych chi'n mynd i'r ymgyrchu y LL fel Adith, dwi chi'n weithio ar gyllid yn gael. Ond mynd o mae'n ein bwrdd, ydych chi'n mynd i'r ddiolch gwybod y paes yn bobl digwydd y dyfanc oherwydd rhai chi'n bwc, ond rwy'n trafnodio ydyn nhw'n mynd i'w gweld. Ydyn ni'n bwc ymdweud, ydych chi'n ddau o ran y bydd diolch cyn i'r ddoch yw gyn ymwysgol o'r Anfield, o'r hyn o'r Llyfrgell Llyfrgell Bramley? Is Anfield history? Do you think? I don't think it's history, but I think the law come down to commercials. I think the new evidence stadium has got elements of its design that allows for concerts. So I think that, you know, with the right management in the stadium, then it could have a good summer season with seven concerts. Yn ymgyrch, rydyn ni'n gweithio'r club o'r oeddiad honno yn ymgyrchol wedi y annan, ond rwy'n gweithio bod rydyn ni wedi oeddiad i'w ffordd ar y maen nhw. Meddwl yn ymgyrchol gwahanol oherwydd mae oeddiad yma, oedd ymgyrchol yn ymgyrchol yng Nghymru, ac yw'r rhaid i ddweud, ac mae'n meddwl am y dyfodol o'r rhaglion o'r newydd. Mae'r ystyried ymgyrchol yn ddweudio'r oeddiad, Paul, Mae'r tŷnodd yn ystod, yn mynd i'r gweld, yn ein ffrustraedd y stailio yn ystod y byddai'r gwrthog i'r clubau byddol Lango Rok, y cyfnod o'r cyfnod o'r ddechrau'r cyfrifau o'r ddweud, ac mae'n gweithio'n ddweud y dyfodol i'r gweithio'n gweithio, dwi'n ddweud o'r 25 eich sefydlu ar y penedig oes ar gweithio. A wedi fyfodol, mae'n ddau yn cyfnod, ond mae'n gweithio ar hyn o'r mwyaf. Cyngoron, dwi'n gweithio'n annun o'r wych gan hyn. Mynd i'n gweithio phobl o'r newyddion, ac bod yna yna fydd yn gallu gallu llwyddon. Well they tell me they didn't want to open it any bens until um the first soccer game. I mean they'll do trially bends for health and safety and more of that kind of thing, they have to do that. But I think it's too important to creas to be a great farewell to the GunstAD park. I think they need to get together and make a big event. You know the last cave for GunsAD Park should be a sy'n golygu'r ysgolio ffyrdd o ymryd i ymddewch, ac rwy'n mynd i'w ddyn nhw'n ddweud yn ystafell. Felly ydw'ch sylfaen ymweld yma yn y dyfodol y dyfodol y 25 yma, a'n rhoi cymryd y stafell hwnnw. Mae'r famy o'r ddechrau yn meddwl. Mae'r bobl ein gweithio. Mae'r bobl ein gweithio, bobl yw'r gennym, yn olygu'r stafell yr ystafell hwnnw wedi'i bod yn gweldio'r bwysig.  Yn mynd i, yw'n mynd i'n mynd i ffordd, is sooner or later the ownership is going to change and at the moment an organisation called 777 Football Group looked to be in pole position, so there's a chance that they will have a different view. If you were walking into Everton now with the man who's just acquired a decent share of it, what would you expect to need to do to start the journey back to greatness? Rym ni'n meddwl, rwy'n meddwl y ll patreon, yna'r credu cyfnod, rydyn ni'n fuddio ar y gwirio'n bwysig fel y game ramio. Ond mae'n wneud hynny o hynny, rhaid i fewn siŵn yn cyfnod digon i ddweud y gyrdwyr gofot yn llunig yn y g beats wagon.  Ac rwy'n ddweud yn ddweud yn ddweud. Rwy'n dweud yn ddweud, mae'n ddweud at y poll a'r ddweud yn ddweud. Mae'r ddweud i'r ddweud, rwy'n ddweud at ymddangos ar gyfer, ac oedd yn gwybod. Felly ydych chi'n gymryd ar y cwerthio'ch ei rôl gynnwys? Mae'r byd yn gyfliadau, felly mae'r gweithio'r cyffredinol? Cael yr ysgol cyndor? Mae, mae'n.... mae'n gweithio'r byd. Wel cyn ddim mneud fydd. 15 April. Ond loswn i ddefnyddio'r armau a gydydd y bydd yn cyfryd. O edrych yn allan o fyw, Paul, unrhyw gweithio fy speakingi ar y mewn dweud o bwyd i'r hyn. A dyfwni'n cael ei ddiwodau'r cyllidion yma o beth am lawer i amlo. A'r dweud â gwrando mustafod yn dda ni'n cael ei siar hwnnw i ddweud gael eu mewn cwmbod. Byddwch hwnnw i ddigon ar eich gweithio. Rhaid i'r ysgrifennu, fath gennym ni'n gweithio i'r gyllideb yn ei ddechrau. Ac ti wneud i'r bramli moll gan y ffyrth gafn waeth, felly i ddweud hynny hefyd sydd gyd wedi'i gín falch a hollol i'r event. I think there are all sorts of things that sourour are evidence to do to capitalise on the value of the stadium. I think footballing, they need to think about what the future is but maybe writing is not a bad idea. If we become a club that makes £100 million a year profit then I think we will be in a good place. Ac yn gweithio'r llwyddo. Ddau'n ddau'n gwybod, Paul. Rwy'n gweithio'r llwyddo, yn gweithio. Rwy'n gweithio'r llwyddo. Rwy'n gweithio'r llwyddo.


Good afternoon everyone. This is Melissa Armo with the stocks which I'm doing this video today for Omar Omar was asking me how in the world do you have time to figure out? What to do or rate the gaps? So quickly I Said watch I don't do it that quickly I take my time because I see the gap before the open so first I wait for the gap to happen A&F had earnings you have no idea if it's gonna gap up if it's gonna gap down You don't have to do anything you wait here is the chart This is the gap this happened See this 7 30 a.m. So it's 7 30 in the morning A&F started selling off So it then at that moment was gapping down. So then you see the gap at 7 30 So you've got two hours for the open to rate it Okay, so you don't do anything until after gaps Then you rate it and I rate it before the open then after that if it rates 20 points or more I trade it on the open and not before so today's Go over This is all the pre-market you're rating the gap here And it's moving. So you're rating it in this area between 15 30 and 14 80 I should mean you have to look at it if the stocks live It's moving in the morning. Okay, so it was a good short. So I shorted it here boom Rallied up again. You actually could have taken more boom. That was it done So you see here how this was done at 10 o'clock for the majority of the move now. Where was the low of this? 1419 I said $14 was a dream target on this today. It actually got within 19 cents to that But I didn't I didn't hold this all the way down here But actually you could have you would have been in it till one o'clock I did not hold it down there Then why because I like to trade the morning and be done and this is all that I need This is all that anyone needs to make a living. This is a nice move in here boom You don't need to hold stuff all day. Do you see how you would have suffered back through a rally? It did keep going it never went through the price We're never down if you took it, but why sit there all day to make an extra 20 30 cents I mean to me it just makes sense to get the train be up right a ways get out You know, I mean you could have taken it off here. I didn't I held through this and got the drop But you know you very well could have you know, just done this so many different ways, but it was a nice short So just to go over Since for Omar you wait to the stock gaps then it gaps then you see it then you rate it and then I take off the pre-market Postmarket information and I trade in the live day and here's what it looks like You just have to put the pre and post market jiggy on your platform. Okay? nice one here It's actually this could keep going down to 14 before the pose But again, why trade in a Friday afternoon at 2 30 in the afternoon when you're ready up and you made money I mean, you know your goals in you take it, you know So for more information if you'd like more information like Omar Or if you have questions email me at Melissa at the stock switch calm Have a fantastic weekend everyone and I'll talk to you soon


Those three sectors, the water, food, and energy, those sectors are key for human survival. The more people we are, the more pressure on those resources, the stronger the interlinkages become. The three elements are really closely linked. You can't basically look at any sector in isolation. If you take the water sector, there's really close links with the energy sector. If you take the energy sector, there's close links with the food sector. And the food sector also, you can't just look at the food sector in isolation. An illustrating example is the energy sector. Now, say if a European government or some government say, OK, we would like to have 10% of our energy from biofuels, then that has enormous implications for the water sector, but also for the food sector. Because the food sector is basically competing or can be competing for the same land resources and water resources. So it takes enormous quantities of water to produce biofuels, but also to produce food. So you see that those kind of links are strong. You see that many decisions that relate to water and to water management are actually taken outside of the water sector. So that is why it is really important that you look at those three sectors as one unit and interlinkages between those sectors. We all know that because of the population growth, because of changes in diets, maybe some people say that, some estimates say that we nearly need to double the food production. But of course, with the same resources, we don't have more resources than we have. So that resources are energy and those resources are water. So we need to produce nearly twice as much food with the same amount of water and the same amount of energy. We do some research in Ethiopia, spate irrigation, they call it. So on a certain time in the year you get these floods, and the rest of the year the river is basically dry. So if the flood comes, the farmers, they try to get that flood, direct that flood into their fields to produce one crop per year. So without that water, without those flesh floods, they would have basically no food production except maybe livestock production. So of course, if the water is on the field, it also infiltrates. So you get also a replenishment of shallow groundwater, and that shallow groundwater is then again used for domestic purposes. So you see a multiple uses of water resources there. I do think it's an incredible important topic and you see that there's a lot of attention also from many countries. So we are quite fortunate here in UNESCO IT to have so many good MSC students, good PhD candidates who all give their little contribution to this huge global problem.


The Databases for Machine Learning and Machine Learning for Databases Seminar Series at Carnegie Mellon University is recorded in front of a live studio audience. Funding for this program is made possible by Google and from contributions from viewers like you. Thank you. Awesome guys, thanks for everyone coming. It's a new semester so we're excited to kick off the new seminar series. For this semester we're doing Machine Learning for Databases and Databases for Machine Learning. In retrospect, maybe I should have called it for AI for Databases because that's more buzzwordy. But nevertheless, the goal is to discuss various systems that are facilitating machine learning AI workloads and then using AI and machine learning to improve database systems. So we're excited today to have Andre Vasenstov, who is the CTO and co-founder of Quadrant, one of the hot vector search engine vector databases in this space. So he's here today to talk about the internals of their system. So for people that are joining us for the first time, the way we want to do this is that we don't want Andre to talk to himself for an hour in Zoom because that's not fun. So if you have questions as we go along, please unmute yourself, say who you are and ask a question at any time. It won't just be interactive for him and not just, again, speaking to the void for an hour. And again, we'd like to thank Google also too for sponsoring us again this semester. So if you're not able to unmute yourself, you can send a message in chat and I'll try to stop and ask it. Okay? All right, Andre, thank you so much for kicking us off in the new semester. It's always a good turnout. So the floor is yours. Please go for it. Yeah, thank you, Andy. And thank you for having me here. I'm Andre Vasenstov. I'm co-founder and CTO at Quadrant. I started Quadrant as a big project about three years ago, basically to solve my own problems at my day job, which later turned into a dedicated company. That's what I'm doing now for full-time. All code in Quadrant goes through my reviews. So you can assume I know it pretty well. I work in the search related fields for already almost 10 years. I started my career as a backend engineer at a web-scale search project, much similar to what Google does. I was working on very low-level components of the search engine, like indexing and storage. Later I switched to more high-level components like query parsing and ranking. I used a lot of machine learning in my job. And at some point we started to use vector search in our projects. And that's basically where I got the idea to build a search engine which is specialized on vector similarity. So what is Quadrant actually? Quadrant is a vector similarity search engine or as our marketing department make me also call it a vector database. But personally I don't really like to vector database and prefer search engine as more accurate one. And I hope in this presentation I will be able to explain why exactly I think so. Quadrant is fully open source. It's distributed under Apache 2.0 license. We have more than 60 external contributors on GitHub. And yeah, please, you're welcome to contribute in Quadrant as well. Quadrant is written in Rust as I think system-level software should be. But yeah, that's a topic for another talk. As for this talk, I propose to cover the following things. First, it's short but mandatory introduction into what vector search actually is and what Quadrant plays like which role Quadrant plays in all of this industry. Second is architectural overview of Quadrant, how it works on a general level, general idea. And I also wanted to clear some terms before we go deeper into vector specifics. And the third topic is especially this vector specifics. So what makes vector databases or vector search engines special? What challenges we faced and how we solve them? And yeah, which technical decisions we made in this process. Yeah, but let's start this first overview of what vector search is. As an approach, vector search is not new. It was used by many big companies for quite a long time. And principle didn't change much since then. The basic idea is that you have some model or we also call it an encoder. Typically it's a neural network. It can convert some input data into a dense vector representation. And those vector representations are also called embeddings. They have very interesting property. A pair of vectors in vector space, which are close to each other, is usually corresponding to objects which are also similar in some sense. In case of texts, it could be a similarity of the meaning. In case of images, it can be visual similarity and so on. And it's actually defined by the model, which type of similarity we want to catch with those embeddings. And also distance function between vectors is also defined by the model. But in majority of use cases, it's just a simple production between vectors. So although this is not new, what is new is how available those ready to use models become and tools around them have become to a wide audience. So now anybody can go and download a pre-trained model and use it without any knowledge of machine learning. So the next logical step actually is to bring this technology to production levels. So not just scientists, but also engineers and software developers can work with it. The next step is to provide the level of convenience and reliability as it exists in such familiar tools, like text and search engines, like traditional databases, and bring this convenience into the vector search world. And that's what actually we do in Quadrant. So let's now take a look at how Quadrant actually achieved this. So here's the top level overview of Quadrant components. Each level in this hierarchy represents some special kind of isolation. On the top level, it's a collection. Collection logically isolates types of data between each other. It's similar to table in relational databases or collections in document databases like MongoDB, for example. After collection, it goes to the next step, which is shards. Shards isolates subset of data between each other. It is guaranteed that shards contain only non-overlapping subset of records. And shards can be moved between nodes. Shards can be replicated for higher availability and so on. And finally, the lowest level of isolation is segment. Segment isolates index and data storage. So each segment is capable of performing all the same operations as a whole collection on its own, but just on a smaller subset of data. And yeah, later I explain why I exactly do the segments and how they work. But before that, a bit more about top level. So you may notice that on the top level, coordinate architecture is pretty standard. We have collections which contain shards. Shards can be replicated and moved around nodes. We use Raft consensus protocol to keep track on this metadata, like where a collection is located, what is the node status, what is the collection configuration, and so on. So all of this metadata is stored in the distributed consensus way. And it's also pretty standard for many distributed systems. And this is actually because quadrant is a typical example of a system built on base principles. And base here stands for basically available soft state eventual consistency. It is typically compared to ACID principles, which are more common to relational databases. So if you want to have an intuitive understanding of what is base and what is ACID, you can think about Postgres and Elasticsearch. So Postgres is a typical example of ACID database. It has very strict transactional guarantees. It cares a lot about consistency of the data. But the scalability of Postgres is actually very limited. It's basically up to the size of the single machine. On the other hand, Elasticsearch is a typical example of base system. It is highly scalable, but consistency guarantees are much weaker. And that's actually why I prefer the term vector search engine rather than vector database. For systems like quadrant, scalability and performance, in my opinion, is much more important than transactional consistency. So it should be treated as a search engine rather than database. Ideally, I think it should not be even used as a primary storage of data, especially considering that the full update of vectors, like due to, for example, a new version of the model, is just a common operation in vector database world. So you need to wipe the whole database and create a new one just because your encoder is changed, right? And it's something which never happens in traditional databases. Yeah, so what happens inside Elasticsearch is a bit more interesting for this talk, because it's actually have specifics related to vector search. On the top level, we see a pretty standard component, right to headlock, which is responsible for the fact that quadrant doesn't usually lose data once it's committed. Yeah, again, it's pretty standard component for any database. What is not standard is the second level of data segregation inside the chart. And this second level is segments. So why do we need many segments in the first place? Yeah, why can't we put just all data into one segment and that's it? And there are actually reasons for this. First reason is mutability. And in quadrant, we actually love immutable data structures. But the simple assumption that structure is only built once and then never extended actually opens room for many different optimizations. Data structure can become more compact. We do not need to jump between different locations in memory. Therefore, less cache misses. All data statistics is known in advance. So based on that, we can perform various optimizations as well. Like for example, we can pre-compute histograms. We can pre-compute data distributions and so on. Of course, in this case, we can also allocate the exact amount of memory we need. So we don't need to worry about memory fragmentation as well. Loading immutable data structures is also much faster because you don't need to perform any kind of deserialization. You just copy a row chunk of memory from disk and that's it. Or you can even do memory mapping even faster. On top of that, we can further compress data using such techniques as delta encoding, as variable byte encoding and so on. And this total effect, total combined effect of this optimization can make immutable data structures an order of more efficient than mutable ones. The second reason actually is trade-off between latency and throughput. And the reason for this is that concurrency of the single request is only efficient to a certain point. And the closer we are getting to low-level index, the less efficient concurrency becomes. So that makes segment a natural unit of parallelization important. For example, if we are dealing with an application which requires a very low latency for a single request, we can optimize CPU utilization by just assigning each CPU core with one segment. And in this way, it will utilize as much CPU as possible in just one request. On the other hand, if we have an application which is dealing with high throughput and it requires to make a lot of parallel requests, we can have a single large segment. And in this case, it will maximize the throughput of the whole system by just serving each request on a dedicated core using the whole segment in read mode. So we have a question from Rohan asking, what is the segment size? Any segment size is any set. What is the default then for you guys? Default is zero. If you don't have any data, it's zero. If you put some data in there, it's growing to the extent where... What we prefer to configure is not the size of segment, but how many segments do you want to have on your machine? And you can go as big as you want if you have enough resources for this. So according to this logic, suppose you have a number of segments, what you're trying to say is that's the max. There could be all storage utilized by one segment itself. That can happen. It is possible, yes. If your application is very read heavy, if you have very low amount of write requests and you prefer to maximize your throughput, yes, it's possible to just join all data into one segment and work with this. Okay. Thank you. There's somebody else. Next question. Is there any advantage to having segments of different sizes within the same shard or collection? Right. So there is no advantage in this because we prefer to have even distribution. But in practice, it may happen because segments are joined like during the insertion of data, segments can be joined together and you always need to have at least one segment which is not immutable where we can put new information into. So in practice, it's not happening, but we prefer to have them as even as possible. Any other questions? Somebody else. Is the segment sizing configurable at runtime? Size, no. The amount of segments, yes. Let's keep going because let's get to the actual vector part. All right. Well, there is one slide about segment management though. So the thing is, we have a lot of segments within the shard and some of them are immutable. Others are just used to insert a new data. But how do we maintain the illusion for the user that actually the whole collection is fully mutable and in quadrant user actually can insert, delete, update any data at any time. So ideally, users should not even know that this segment exists. It's pure internal thing. And in order to solve this problem, we need to actually solve two problems. The first is how to update data and immutable data structure. And second is how to even obtain the immutable data structure in first place. So the first problem is solved by simply employing a copy and write mechanism. So whenever a user inserts a new data into or changes data in the immutable segment, we just copy this piece of data into immutable segment mark. It is as deleted in the old segment and everything works. The second problem is a bit more complicated because we need to perform long-running optimizations on a segment. So index building is quite long. That's why we need to keep this segment available for read and updates for the user. At the same time, we need to build index from it. And in order to do it, we use so-called proxy segment, which is a special type of segment that wraps in one under one interface. It's a segment which is currently being optimized. And it also holds a list of modifications. It needs to apply to resolve conflicts which is happening when you copy data from old segment into a new one. There's like special data structure which manages all these insertions. And when optimization is done, it converts back into just regular segment, into pair of segments, actually, into optimized one and into a small copy and write segment which just becomes immutable segment.  I intentionally am not going to describe how exactly each of them works on a low level. Instead, I will focus on vector index mainly. And the reason for this is that the choice of concrete implementation just depends on configuration. For example, vector storage, we have at least three different implementations of vector storage in Quadrant. And it's quite possible that in recent future, we will add the fourth one. And it's not really important. Quadrant works able to work with any abstractions. It can be just files. It can be memory storage. It doesn't really matter. But this matter is vector index. And let's finally talk about the main component of the vector search. The vector index, there are two main traits that distinguish vector search from traditional indexes like inverted index or P3 or stuff like this. First, it's approximate nature. It says that it does not guarantee that the result will be exact or even that the result will be the same for the same underlying data if you build the index multiple times. Because the result actually depends on the order in which you insert data into the index. So it's quite fundamental. And the second is that any vector can be the result of any search request. In other words, any documents in your collection is somewhat similar to any other document in your collection. So it's impossible to draw a clear line between relevant and irrelevant documents based solely on the vector similarity score. And of course, there are a lot of different approaches how to implement vector index, but no matter which approach you choose, all of them will have to deal with these properties I just described. And actually it breaks many assumptions you made in a traditional database. And therefore I believe it actually requires a special treatment and special dedicated architecture around it. In Quadrant, we use so-called HNSW index. Here HNSW stands for Hierarchical Navigable Small World. But yeah, it's a pretty complicated name, but I will try to explain in a very simplified version of it just to provide some intuition about how it works. So internally, HNSW appears as a proximity graph. That means that each vector is represented as a node in a graph, and those nodes are connected with some number of closest neighbors. So some number of other vectors in the database. And search in the proximity graph is performed in greedy manner, meaning that on each step, we choose the closest node to the target and then repeat the search step with this new selected node. The process is repeated until we can no longer improve the distance between node and the target. And of course, there are no guarantees that this search will result in the absolute closest vector. That's why it's called approximate. But we can control the precision and trade off precision and speed of the search by changing the beam size parameter of the search. Yeah, but of course, HNSW index brings its own challenges. First of all, building time. Insertion of a new vector into the index is approximately two times more expensive than just searching index. And it's also very CPU intensive on its own. Therefore, if we do want to index and do not affect our other processes, like if we do index and search at the same time, it's necessary to have a dedicated thread pool just for building index in the background. Or ideally, we might even want to move this index building process into another machine completely. HNSW index is not only CPU intensive, but it also have a random data access pattern. That means that in every... That means every... It's very sensitive to the latency of the underlying storage. And techniques like prefetching, like block reading are not really efficient with HNSW. That's why it usually requires a lot of RAM and doesn't really work with this disk. And moreover, the pattern is not only random, but it's also sequential. Remember, we go from one node in a graph to another. That means that we cannot... We cannot do efficient parallelization of the search and its performance is mostly limited by the latency of the storage rather than throughput. To overcome these challenges, the quadrant we do the following thing. So the solution is to use a compressed memory representation of vectors and use it to generate selection of candidates. For instance, one of the latest addition to the quadrant engine is binary quantization. It allows us to compress vector to a level where a single dimension is represented by just a single bit, which gives total of 32 times compression for the vector. And on top of that, it allows us to use very fast CPU... Basically, it allows us to compare vector in just two CPU instructions, bitwise XOR and popcorn. And it works especially efficient with large vectors, like the one which provided with open AI models, for example, where we have 1,500 dimensions for a single vector. Yeah, after obtaining this list of candidates, we can rescor them using the original vector and return the final result to user. And it's important to know that this rescoring process, unlike traversal of the HNSW graph, actually can be efficiently parallelized as we already know all the offsets, all the IDs of candidates. So we can leverage here the synchronous IO and take advantage even of the slow network mounted disks with huge latency. So you said it's rescoring, so you take the quantized vectors and then what do you do it against the original vectors again? Right, so we do search. Like we do usual search using HNSW using quantized representation. So it's very fast, very memory efficient. And then after we obtain, like let's say, top 100 results, we do rescoring using original representation. We do need to fetch those 100 vectors from disks, but while we do this, we know all the offsets, so we just make a parallel request to disk with asynchronous IO or something like this and just get all the original vectors pretty fast. Yeah, so you're compressing the vectors into the quantized form. That makes the index look faster, but do you still have to do the final comparison to see whether true rankings should be? All right, there's a bunch of questions in the chat. So the first question, is this equivalent to what disk A&N does? I don't know what that is, but assuming it gets... Yeah, disk A&N is a bit different. So I don't think disk A&N does any kind of quantization. There are other implementation which does. So I don't think this approach is somewhat like revolutionary, but yeah, it's parked into the box where you can use it without any additional configuration. You don't need to pre-quantize your data, your vectors you just upload everything in quadrant and it just works out of the box. So quadrant takes care of this quantization itself. It takes care about the scoring and you get the result. The next question we'll hold off to at the end. They're asking basically how many vectors can you put on a single box? Well, that's an arbitrary question. We'll come back to that later. Keep going. Yeah, all right. So another challenge which is associated with HMSW is a requirement to combine vector search with additional filtering conditions. For example, you might want to search for some kind of item in e-commerce store and the price of this item should be less than $100. And we want to search it on a specific location like near you, find something near me, some stuff like this. So those additional conditions are really necessary for real-world applications, but vanilla implementation of HMSW or any other NN algorithm just doesn't have this. And in some publication, you might find that there are two ways to solve this problem. One is either post-filtering or pre-filtering. In post-filtering, it is suggested that we can perform a regular vector search and then apply filtering conditions later on top of the result and exclude those results that just do not match filtering criteria. We might need to repeat this process several times until we obtain the required number of results. This approach is quite simple to implement, but unfortunately, it's very inefficient, especially if the filtering condition is very strict or if it's correlated with vector similarity score itself. So this approach basically risks to either turn the whole search into linear scan or return incomplete results eventually. Another approach is other way around. It suggests, I mean, pre-filtering. It suggests to generate a list of candidates and based on this list of candidates, we can perform a vector search. And the problem with this is that generating a list of candidates on its own might be a very expensive operation. Worst case scenario, it might require to check condition for half of the vectors in the collection and that can significantly increase the search latency on its own. So what we propose an approach that is used in quadrant we call in place filtering. That means that filtering condition is checked during the graph traversal. It did require us to make some custom implementations, custom adjustments to HNSW. So we don't use vanilla implementation anymore. We use a custom implementation quadrant. But in this way, we can ensure that we only need to check filtering conditions the amount of times that you actually needed to perform the search. And like it looks like a problem solved, but unfortunately not. The problem arise when the filtering condition is so strict that the graph become disconnected. That means we cannot longer find a path between the entry point in the graph and the section which actually contains the desired results. And in mathematics, there is a whole field of study called percolation theory, which is dedicated to this special type of problem. And for large random graphs, this theory actually gives us surprisingly simple equation that defines how many nodes should be removed from the graph to make it disconnected. And equation is simply one over K, where K is the average number of connections per node inside the graph. In other words, let's say if we have 10 connections per node, after removing 90% of nodes, the graph will become disconnected. And yeah, let's see how it looks in practice. In practice, this plot shows the precision of the search in relation to the fraction of vectors being filtered out. And you can see that after a certain point, the accuracy drops almost to zero. And there are several experiments which demonstrate that this effect actually depends on number of connections per node. Indeed, it's very close to theory. So what we can do about it? To address this problem, we can leverage the fact that filters we want to apply to the search are not actually random. In most cases, these filters are based on some metadata or payload associated with vectors. As you can remember, we have a price next to an item and we know price in advance. And we not only know the price in advance, but we also know that what price is associated with which vector on the index building stage, because we use immutable data structures. And what we can do is to build additional links that we can generate based on the existing pilot values and possible filtering conditions. For example, if we have a pilot with a keyword field, and we can build subgraph for each value of this field and then merge this subgraph for the keyword into the main one. And that's how we can create these additional links. And these links will ensure that when we apply a filtering condition with this keyword, our graph will always stay connected. No matter how many, how strict the search is, we can guarantee this. And this approach... A quick clarifying question. On the non-vector components that metadata, is this search on that guaranteed to be exact or is that also approximate? Yeah, if you are talking about this type of queries... So it's a combination of vector search and filtering. So you are performing approximate search within this condition. So it still stays approximate, but we can guarantee that the condition are satisfied. Got it. So at a high level, you're probably vectorizing those additional attributes itself and just creating a gigantic link to index number. Now, vector stays the same. We keep a payload next to the vector. It's stored in the quadrant. We perform this search. And on each step during the search, we check the condition. So we don't go into those nodes, which are filtered out. So it sounds a little bit like post filtering, right? You'll still do approximate nearest neighbor search following the HNSW graph on the vector and only on the additional metadata you'll do an exact search. Am I getting that right? No, not exactly. The difference with post filtering is that we apply it, not after we generate the candidates, but during the search. So in this case, we can guarantee that we will return exact amount of results we request and all of these results will satisfy the filtering condition because we restrain our search procedure with this condition and we do not let the search process go into area where the condition is not satisfied. Okay, thank you. Sorry, a quick follow up on that, I guess. Would quantization affect the filtering? Like would we accidentally filter something that could have been satisfied if we had the uncompressed factor? No, no quantization is only affecting the precision. It may affect the precision of vector search itself. Filtering is completely untouched. Okay, cool. Thank you. Any other questions so far? So we were talking about building the additional links in the graph. And, yeah. So the good side of this approach is that it doesn't actually increase search complexity. Even so, we increase the size of the graph, we can still perform the search only using the original links where it's needed and utilize extra links only when original links are being filtered out. So the total complexity of the search is not affected. We can index as many additional links as we want. We can index many additional fields. Yeah, that's another point, actually, is that this approach is compatible with multiple fields being used for filtering at once. Because when we do that, we merge all the subgraphs into the main one and we basically can deduplicate links at this stage. So only a fraction of additional memory will be actually required for this. And search speed won't be affected. What matters? So Sid asks, does quadrant do any cardinality estimation to determine query plans? Yes, it does. Unfortunately, I don't have slides for this, but I'm happy to answer those questions after the main topic. Awesome, thanks. I had a quick question. If I had something like customer ID where I know those things are always going to be separated out, would you recommend doing different collections or is there something else to recommend filter? Right. It's the most frequently asked questions in our Discord community. By the way, feel free to join it. I'm answering questions there as well. So what quadrant can do is to build HMSW graph only based on payload. So we can skip building this whole main graph for all points and only build subgraphs for the specific user IDs. In this case, we can still have search performance within user, right? It's all very fast. We can still have full scan ability to look through the whole collection. And at the same time, we do not have to spend as much resources on building vector index for all points altogether. So yeah, it's like a solution which allows you to do like semi-segregation of data. They will be still stored in one collection, so no overhead for creating many collections. And you are able to perform searches on the subsets of your data. Anandri, feel free to defer this question if you're going to talk about it later. Are you going to describe what's the complexity of creating this HMSW index, both as the number of records grow and as the dimensionality of each of the vector grows? Right. So I will start from the last question. Dimensionality of vector affects how fast you can compare a pair of vectors, right? So it is basically linear in this case. If you have, let's say, 1,000 dimensions, it will be two times slower than 500 dimensions. Complexity of graph search itself and building of the graph is like there is no exact estimation for this, but you can say it's approximately logarithmic. And the process of indexing actually involves search. So in order to insert the new data, you need to first search for its neighbors and then perform changes in the graph. So overall, yes, it's quite expensive to build large indexes. That's why there are some optimizations that we are trying to do on this as well. That's why you might want to have multiple charts and multiple segments to limit this overall size of the segment. Yeah, that's a good point. And what you just said, can these links cross across segments or sharps? Sounds like not across, but across segments. No, no, these links are isolated to one segment. One segment. Okay, thank you. So what we do when we do a search, we basically ask each segment individually and then merge the result of the search. Similar approach which is used in all other, even text search engines, it's pretty standard thing. Yeah, but yeah, thank you for asking. Maybe it's not clear for somebody who's not deep into this topic. Yeah, so one, yeah. Sorry, can I ask a question about searching across segments? So I was wondering, are the segments just random? Are they based on like some sort of spatial partition? So points in the segments are likely to be close to each other or like how's that approached? Right, so in our implementation, segments are completely random. Each time we insert a point, we just toss a coin and put it into first segment which is appendable. Yeah, we do not do any kind of clustering inside. Mostly because the clustering actually depends on type of factors and it depends on model and we cannot make these assumptions in advance. It might be a good approach if you're building like a dedicated system for some application where you know which encoding model you're going to do, but for general purpose, it's unfortunately not really working. Hi, sorry. How are these expected filtering conditions determined? Are they predetermined in a sort of like prepared query fashion? Yeah, so what we have in advance is we know what payload user uploaded into quadrant along with the vector. So usually it's in a form of just JSON document and user can specifically say which fields should be indexed. Like let's say we have a product, we can e-commerce store, and we have a long text description, name, price, and category tag, something like this. And it's usually only price and category tag are going to be used inside search filtering conditions and that's why user can specify that those two fields should be indexed. And once they are indexed, we know that they should be included into this process of building additional links in HNW. Cool, thank you. Anything else? All right. So one important thing about this additional payload and associated payload indexes is the data type of payload does matter. But in most cases it is possible to come up with a strategy to cover filtering conditions with a subgraph. It's especially interesting that we are talking about, let's say numerical field where we don't have exactly a keyword which defines a strict subset of points. But in this case we can, for example, what we do in Quarant is we build a subgraph for overlapping intervals. We know that for some intervals, which interval covers how many points and we know this minimal threshold of how many points should be in the graph. So we build overlapping intervals and we build these subgraphs with additional links for those intervals. And also we can do the same with, let's say, location data where we have geo-coordinates. So latitude, longitude, we encode it into geo-hashes and basically build these additional graphs for overlapping geo-hash regions. And yeah, the result of this is that we can close the gap in filtering precision from two sides. From one side we close the gap by introducing these additional links. And from the other side, we increase the connectivity of the whole graph. And in most practical use cases, the drop of precision is not noticeable next to the fundamental approximate nature of HMSW itself. It is just like a noise and no signal. Well, actually it's all topics I have presentation for. Just for those who just joined, here's a short summary of what I try to cover. So first of all, vector search is only getting started. I believe that there are many more interesting use cases beyond just text search or memory for chatbots. Second point is the quadrant is a search engine and should be treated as such. Architecture of search engines is fundamentally different from the architecture of databases. And it should be taken into consideration when you are designing your own application. And finally, vector search index is a pretty specific component and it requires special treatment even for such usual operations as filtering. So what is out of scope of this presentation, but I'm happy to answer questions about it and even beyond this talk is query planning and payload indexes. How do we organize it inside the quadrant? Interesting topic is dynamic search limit per segment. So as I mentioned, we have multiple segments and in order to get the whole search result, we need to go into each segment and ask for some amount of results in each of them and then combine. But if we want to search for the very large limit, it might be expensive to do. We have some optimization for this. And yeah, more quantization. We currently support three versions of quantization. It's binary quantization, the most recent one, scholar quantization, product quantization. We help all of this. Yeah, that would be all. Thank you. And we are happy to answer questions. Awesome. So I will applaud behalf of everyone else. If you have any questions, again, unmute yourself. We have about nine minutes left. Go for it. Maybe I'll ask a question. Andre, that was a really nice talk. There are two different grades which people are going about, putting vector search and data-rich applications. One is where you start with some sort of a relational code like PG vectors and you put it in Postgres. The advantage there is that if your data application needs to research on these additional attributes, regular columns, you can get very deterministic semantics on the search for that type of metadata. And of course the disadvantage is on the vector search, you can't build these quote-unquote global indices like HNSW. Do you have some thoughts on what kind of applications from your perspective fit which style of embedding vectors along with relational type of search? Or you think this approach, your way of doing it, or do you think there might be one way that basically captures everything that will emerge? So regarding Postgres and relational databases, it's what I was talking about, difference between base and ACID type of systems. So Postgres, as I mentioned, is a typical ACID database. It have very strict transactional guarantees. It have very strict consistency. But the scalability of Postgres is very limited at the same time.  Well, probably vector search and vector database is not a good choice if you're building some kind of banking system where you have money transferred, right? At the same time, maybe elastic search Underneath that, I have a question. If I have an application in which I need to do both relational search and vector search in the same application? Well, if you, by relational search, you mean some kind of joins? Yeah, not even joins. Imagine I have a big table. I want to search on different columns in different ways, depending upon the user query. And I've got a vector column that is part of that search. So my search may involve finding everything where the student ID is in this range, the age is in this range, and then there is some vector representation of maybe some descriptive stuff about them. I also want to search on this. I want to mix exact search and approximate search together in my end application. Well, yeah, that's what exactly these filterable queries are doing in Quadrant. And well, the answer for this is, it depends if you are satisfied with scalability limits which exist in Postgres. In most cases, I think Postgres will do just a full scan if you try to combine these relational filters and the semantic component. So if you are satisfied with scalability of Postgres, yeah, you can do that. But also I mentioned that search engines are rarely the source of truth in your system. So what usually happens is you have your Postgres and you also have some additional indexing engines. You can have elastic search for text. You can have Quadrant for vector search which do not hold maybe even very up-to-date data, but they can perform specific types of queries very, very fast and scalable. And you can still have Postgres in your system as well. So it's not excluding each other. Thank you. Avery, you want to ask a question? Yeah. I was curious about this addition of new links things. How do you decide how many new links to add? And also in practice, how much does this blow up the graph? I was kind of curious if you have any predicate that is any attribute that's even relatively high, like 100 values, let's say, you would end up adding a lot of new edges in order to make it work. Yeah. So let's say he's hiding his name, I'm all despondy. The extinguished data is a fessor. Just as AD. All right. So we allow users to configure this value, this number of additional links in the configuration of collections. But by default, we just use the same amount as the original graph have. And overall, it's not usually a big problem because as I mentioned, those links are being duplicated. So if some links exist in original graph, it won't be explicitly added on top of this after the merge process. That's the first thing. Second thing, overall, HMSW you have a special heuristic, which allows to exclude redundant nodes and so on. So even if you try to configure HMSW, let's say 1,000 neighbors for each node, it won't actually build graph with 1,000 neighbors because of this heuristic, which will just prune redundancy. Yeah, it's not included into my slides, but it exists there. Just for simplicity, I try to avoid these very deep descriptions of HMSW. Thanks. Avery, you want to go next? Yeah, sure. So I want to ask how do you think of HMSW plus convolution compared with the scenario? Because as we all know, currently, the scenario also has some variables involved in filtering. Sorry, I didn't get the question. She's asking how does your approach with HMSW plus the quantization compare with Microsoft's disk ANN? Yeah. All right. Thanks. So as far as I understand disk ANN implementation, it doesn't use quantization in the first place. So the idea of disk ANN is that you have a data structure similar to HMSW, but without any hierarchy. So it's a plain graph, much like I have on my slides here. And the difference is not exactly in how you search in this graph, but rather how you build it. So HMSW assumes that when you insert a new point, you just create a new links for this point in the existing graph. But disk ANN implementation, on the other hand, it works in other direction. It builds a fully connected graph and then prune it. And I may be wrong here, but as I understood, disk ANN is something like this. So disk ANN doesn't use any kind of quantization. It doesn't have in memory storage. And actually, if you think about it, here there is no limitation on which kind of indexing algorithm we are going to use. We can use this quantization plus over sampling thing with disk ANN approach as well. So it's something on top of vector index. It's agnostic to vector index type. All right. Alessandro has a question. Yes. First of all, thank you for your talk. My question is from the point of view of a possible user of your application, so not too much technical in the details of the engine. I can get it for image and maybe text, but do you have an experience with time series data? So maybe to check a similarity within time series and what are the challenges in this field if there are any? Yeah. So, well, in Quarant, we do not exactly work with embeddings. Quarant assumes that embeddings are already made from some external model. So I would like to say the short answer to your question is, if you have a model which can translate time series data into a vector, then it's going to work with Quarant as well. A bit longer answer to these questions probably going to involve like, it's going to depend on what type of time series data you have and how can you work with it. The most common thing which I have experienced with and which is a bit closer to time series than just text is user events or user behavior patterns. So for example, if you have a history of user transactions in like a banking system or something like this, you can build a model which tries to predict next transaction and it can do this prediction through the vector embedding. So basically, you will have a model which can represent a series of actions as a vectors and an event in this system will also be vector. So it's very close to, for example, how Word2Vect works. Word2Vect tries to predict next word in the sentence the same way we can train a model which can predict next event in a sequence. So we can use this vectors from this model in Quarant in vector search. Okay, thank you. One last question. So this is the question I'm going to ask all the speakers this semester.  Like if you had a magic wand to fix one thing, what would that be? In our system, making it cheap to serve billion vectors. It's one thing like our milestone for the engineering team, I guess, for the next year. So right now it's quite expensive. You need to have even this binary quantization and stuff like this. You still need to have a lot of memory. It has a lot of problems with this random access and so on. So our main challenge is to make it cheap.


Okie dokie thought I'd share this little vid with everybody as you can see I got a DS 1054z this arrived today 35 minutes ago I put it on I stuck some probes in the ground But a motor got up here just to see what I'd get and the waveform that you can see is a Well, it's nowhere near as good as the one that I was getting but never mind That's not what I'm trying to show you what I'm trying to show you is this Kind of get a focus No, my pizza John camera There we go. There you go. Now you can see that carrying Yeah, you can see my software version you can see the What it is and you can see at the top of the screen like you can see what my ride go actually is so there you go Very easy to do. Let me just A quick jump to the screen very easy to do as you can see there's the there's the info and what you want Is that baby there? That's the link. I Don't know that's good for you to see But that's the link you want. I know there's my serial number. There's the all information And you just go down there just ensure that you are you choose all options when you look down the list and make sure you put That in the option. So Df ds fr Yep, and there's the link So good luck folks. It's brilliant. I've got the latest service pack And firmware version of mine and it still works on that. So it should be great for videos See you later