Mi yw Sarah Turner, rydyn ni'n gyfnodd y dyfodol ar gyfer y Fyglwyr yma yn Y Llywodraeth Cymru. Rydyn ni'n gweithio'r cyfrifodol i'r ffordd o'i bod yn ymddangos i'r dda'i. A'r hyn yn ymddangos i'r ffyrdd yn ymddangos i'r cynllun o'r symposiwn, Lucy Steed, dyfodd Llywodraeth Llywodraeth Cymru, rydyn ni'n gyfer y front i'r cyfrifodol i'r grwm. a Hemaad Nosha, sydd yn gyflawni gyfnodd ar y poriad dynol a gyfermiannol. Mae Cyngor Llywodraeth yn ymgyrchu Llywodraeth Llywodraeth Cymru, ac yn ymgyrchu Llywodraeth Cymru. Ac yn ymgyrch yn ymgyrch ymgyrch, ac yn ymgyrch am ymgyrch, ac yn ymgyrch ymgyrch yn ymgyrch, ac yn ychwaith gweithwyr, ac mae'n dweud o'i felon rhuniau a'i ddweud yn gweithwyr yma ac yn y gallu'r gweithwyr amddangos. Yn i meddwl, rhywbeth yn ddefnyddiad Caith Jessen, Natasha Howes, fries Victoria Wilson, Alastair Huston a'na ar ddysgu'r cymdeithas yng Nghymru yn ei hynny'n helpu'r synthoseon. Y回odd Lyfiad Lwyddon haf o fwy ffiannig ddau wrth yr ogylched, mae'n arwynt i'r parwydd iawn, gyda gweld, criatwyr ac ysgol. Fy na gennym nhw'n gwybod ymddangos ac yn ymddangos, ac yn ymddangos y cyfnod o'r gweithio'r gwasanaeth. Mae'r gwasanaeth ymddangos i'r gwasanaethau yn ymddangos cyfnodol, mae'r Llyfrgellol Llyfrgell, mae'r UAL, mae'r University of Manchester, ac yn amlwgais i'r cwlaeddau yn y ddechrau, David Butler, mae'r ddechrau'r Llyfrgell. Mae'r ddechrau'n gweithio'r Llyfrgell, yn yw'r cyfnod oedd yn ymgyrchol i'r ffordd. Rwy'n dweud i'n gwneud yn gwyloedd o'r Cyfnod i'r ein hynny'n gilydd iawn. Mae'r ddafiau, ac mae'n gwneud eich gweld yr argyrchu yn y ffordd o'r cyffredinol yn ymgyrchol gan yw'r gwahod. Rwy'n ddweud a'u bod yn ymgyrchol i'r cyffredinol, rwy'n gwneud i'n ddweud a'u cyfnod o'r gwahod o'r cyfleoedd ymgyrchol o'r cyffredinol ymgyrchol i'r cyffredinol with Hermar and Kate, and hear their ideas and have these amazing moments of punctuation. To really think in new ways about the exhibition. Exhibitions in many ways are not just the end points of project so much incredible work goes into them and getting them ready. But their really launched their springboard for further thinking and research and hopefully provoking and prompting new thinking, new narratives, and new ways of practice. ymddangos iawn, mae'r modd i'r gweithio'r LYC yn ysgolwyddiadau, a rydyn ni'n gwybod i'r gweithio'r LYC yn ysgolwyddiadau. Mae'n gwybod i'r gweithio'r LYC yn ysgolwyddiadau i'r hysri ac yn ysgolwyddiadau i Llywodraeth a Llywodraeth, i'n gweithio'r LYC yn ysgolwyddiadau i'r gweithio'r gweithio'r llaw o'r cymrydau cymrydau, a dydy'n mynd yw'n mynd i'ch ddweud â'r Llywodraeth i'ch ddyn nhw i'ch ddweud Llywodraeth's ysgolaf, i'ch ddigonio Llywodraeth's cymrydau i'ch ddwyltau yw'r llaw, bwysig i ddweud hynny'n ar hyn yn ysgolwyddiadau. Ysgolwyddiadau'r LYC yn ysgolwyddiadau i'r number o prosiectau. Mae'r Llywodraeth yw'r modd i ysgolwyddiadau i'r Llywodraeth i'r Lywodraeth i ddechrau hyd nu, ac yn ystod o'r projektyn Lundin, ychydig yw Hymard i'r Cogonfyn yma ar y Cogonfyn. Mae'r projektyn yn gweithio gweithio o'r tari gweithio ar gyfer y brithys arddangos a'r hynny o'r hynny o'r hystrys i gael i'r hynny o'r prydys i'r cymdeithas o'r bwysig a'r hynny o'r hystrys o'r rhwn psychosig yn Brytyn. Dyna chwarae ymwneudol wedi gwneud bydd y cwysffordotau i会 yaptig ac bydd yn gallu bod meddwl yma yn y DU они nad yw arddangos i ddim yn leirio Cymru i gynhyrch angen i i ddweud ond mae'r dynig fi whitesbeth o blaen ac robotics am y brifysghau ac gan Là'r retiren ac nowelarn nhw i dweud ac iechid ar Pygysig You are part of this collaborative research project and programme of thinking together inspired by an exhibition. I want us to think about our sessions with their treos, the two papers and the respondents, as something of an LYC three-way ping-pong table. We are going to have a playful bouncing of ideas between us all, and there will be plenty of breaks within the programme. We encourage you to go and explore the exhibition in those, to watch Helen Hett's amazing film and to share ideas amongst yourselves. Now I am going to hand over to my collaborator and master ping-pong player, Hermad Nasser. Thank you, Sarah, and I think there will be bats involved at the table upstairs today. I am going to start very briefly with a couple more things. First we just acknowledge Christopher Payne, I did see him from the University of Manchester, he was very much part of the team and put him together at the symposium, and also thank the many, many people who made not just the exhibition, but particularly this paper possible, Nick Sawyer, the team at the Taipei Fine Art Museum, who were so generous in sharing their research, Mei Qing Fang, Wei Yu, Kai Wei Wang, and Jeanette Martin and Stella Hawkeard at the John Royden's Library. Last but not least, Sarah Turner for her valuable feedback on an earlier version of this paper. As I see around, maybe I think only three of you have suffered before, so apologies for that, because you will see some familiar images, but I will make a start. Now in the last few years the work of Chinese-born artist Lee Wan Chia, 1929 to 1994, has featured in two very different exhibitions on two continents in two national institutions that could both claim him as their own. In Taiwan, the Taipei Fine Art Museum did just that, with a posthumous retrospective that positioned him as the father of conceptual and abstract art in Taiwan. It traced his practice over five years, over five decades rather. In Britain, however, Lee remains largely unrecognised. Though he was home from 1966 until his death in 1994, his work was a minor presence in Tate Britain's migrations journeys into British art in 2012, and a recent display at Tate Modern was only slightly more expansive. His work was entirely absent from the Tate Britain exhibition Conceptual Art in Britain 1964 to 1979, and is unaccounted for in the generally circulating institutional and academic histories of art in Britain. This is at least partly due to the distinctive genealogy of Lee's work that resists attempts to slot it into a singular label. He drew liberally from modernist, Zen, Buddhist and Taoist practices. To explore ideas of space, life and time, the initial vehicle for his exploration was the point, the origin and end of creation that was so beautifully evoked in two of the performances yesterday, both from Madeline Huigast and Carol Cee and Bettina Fung. Originally a spot of colour or a mark in monochromatic paintings and reliefs, it eventually took the form of magnetised objects or toys, as these call them, inviting active audience participation. Now this sense of play and participation was taking new forms by the time of his Golden Moonshow in 1969 at the Listen Gallery. The retrospective at the Taipei Fine Art Museum and a growing engagement with artists associated with Signals Gallery has sparked new interest in Lee's work, with a clutch of recent exhibitions in London and Taipei. But despite this more recent interest, art history is yet to seriously engage with arguably his most important work, the remarkable space he founded in the Cumbria countryside. The LYC Museum and Art Gallery was located in the village of Banks, a stride Hadrian's Wall. Lee Wan Chia's initials gave the museum its name, LYC. Between 1972 and 1983, the museum showcased the work of more than 320 artists, from local figures, Andy Christian, Suzy Onnath, to totemic national artists such as Paul Nash and Barbara Hepworth. Contemporary artists now of international renown, so of Leisha Clark or Andy Goldsworthy, but then barely known in Britain. You'll recognise this picture from yesterday.  The networks and practices that the LYC Museum enabled and enriched have yet to be studied widely, for example, his friendship with a concrete poet and Benedictine monk Don Sylvester Houdard, or the pioneering sound artist Delia Darbyshire, Lee's assistant, and briefly partner at the LYC Museum in 1976 and 77. Friendships and the expanded notions of infrastructure, they often become part of, are growing areas of recent scholarship. That can productively illuminate any consideration of the LYC, and I'd particularly like to point to Leila Gandhi's charting of the politics of friendship in her book, Effective Communities. And in her recent essay, Infrastructure as Form, Karen Zitzewitz argues for the need for us not to separate the analysis of art from the activities from which they emerge to do so, she contends, would attribute artistic production exclusively to the artist and ignore how ably a network art infrastructure distributes agency among its elements. I would argue that the LYC Museum is an exemplary site from which to explore both these formulations, the question of how friendships and form shared practices generate work and socialized narrative, and how the LYC itself functions as a kind of infrastructure. As Sarah mentioned, the LYC Museum has been evoked in a stylized reconstruction and is very much at the heart of the exhibition that hopefully most of you have seen upstairs. This paper is a series of notes and speculations towards a larger collective project of which the symposium is an important part. It explores these ideas of friendship and, to a lesser extent, infrastructure through the lens of three works in the Speech Acts exhibition. All three works are participatory and have to a different extent collective authorship and a relationship with me. It attempts to read the traces of friendships in objects and situations. The principal artwork is the LYC Museum and Art Gallery itself. The second is the video work point in time by Madeline Huicas that you can see and Elsa Stensfield that features Lee as a participant and you can see in the exhibition upstairs. The third and final example is the newly commissioned film Space and Freedom by Helen Petts that I hope you will get the chance to see over lunchtime today. Now the LYC Museum has often been positioned as Lee sacrificing his own practice while he gave his and I quote attention to others. But I would like to place the LYC Museum in an artistic trajectory that saw Lee moving from static objects to the production of work that invited participation from audiences to animate the work. The museum started life as a set of dilapidated farm buildings that Lee acquired from his friend and neighbor, the painter benefit Nicholson. The LYC Museum ran an exhibition program as we've already discussed of prodigious range and eclecticism. It reflected Lee's circuitous cosmopolitanism and his commitment to art as a mode of experimentation. For Lee was also a poet, a designer maker and a curator of art and social interaction. Born in Guangxi, China, he moved to Taiwan in 49 where he was part of the Tong Fan group of artists. Experimenting with abstraction, in 62 he moved to Bologna where he was associated with the Puntau group of artists. An invitation to show at the Signals Gallery from David Madala brings him to London in 1966. There he had multiple solo and group exhibitions at the Listen Gallery between 67 and 70, showing alongside artists such as Ken Cox, Mira Shendo, Ian Hamilton Finlay, Derek Jarman. But London's regard for Lee was not wholly reciprocated. A trip to his friend Nick Sawyer's family house in Cumbria, Boothby, for Christmas in 1967 saw him settle in nearby Bangside, opening the LYC Museum in 1972. The museum consumed Lee. He built it himself, undertaking all building, plumbing, electrical work. It hosted four new exhibitions a month, each accompanied by a catalogue that he designed and printed. Apart from galleries, the LYC Museum had an arts room, a library, a performance space, a printing press, a communal kitchen, and a garden. It hosted rug-making workshops. Children played in its courtyard. It was an open space for the multiple possibilities of art. The artist Sheila Wakeley, whose work is also in our little LYC hut upstairs, exhibited at the LYC in 1979 and saw the museum very much as a work of his. It was an example of social practice before such a thing was named and tamed. After its closure in 83, it became the site of Lee's experimentation with hand-tinted photographs. Lee's establishment of the LYC Museum in 1972 has been called an impulsive, intuitive move. I read the LYC's formulation as a central progression of the experimentation that Lee was engaged in during his time at Boothby, 67 to 71. Boothby was Sawyer's stepfather's family house. Wilfred Roberts was a farmer turned radical liberal politician who Sawyer recounts as, my quote, playing chess with Wyndam Lewis, getting drunk with Dylan Thomas, and dining out with Stalin. Robert's elder sister was Winifred Nicholson. Wilfred and Winifred were grandchildren of George Howard, the 9th Earl of Caillir, and a painter, as his obituary notes, of pre-Raphaelite tendencies, and a younger member of the circle of William Morris and Byrne Jones. So Boothby was the repository of unusual artistic treasures. A bombard hung in Sawyer's bedroom. Sawyer recounts coming through hand-painted William Blake manuscript as a child, and being surrounded by artworks by Moore, Rosetti, and of course, Winifred Nicholson. Now Nicholson moved from Paris to Boothby early during the Second World War and made it her Cumbrian base for the next 20 years. She painted views from the house as well as its interiors. And on arriving at Boothby on Christmas in 67, Lee struck a friendship with Wilfred and Kate Roberts and was offered a large studio room in Boothby at nominal rent. The very room previously occupied by Winifred Nicholson, and those very views also inspired Lee and found voice in his poetry. And I'm sort of just particularly draw attention to this sort of the bold bit. So I like Brampton for its make me to love this world again. It sort of highlights what Lee had spoken about earlier of coming to Cumbria in search of space and freedom and managed to establish a livelihood through gardening, painting, decorating, which allowed him to experiment in his studio at Boothby without the need to sell to survive. And the two self-organized exhibitions he held in Boothby evidence a growing interest in exhibiting environmental and participatory works. The first was held in October 68, the second in June 69. So both before the listen gallery show that showed you an image of earlier. Now Richard De Marco's digital archive offers a set of installation shots of the first Boothby exhibition. The photos show elaborate arrangements of Lee's circular disc, many with multiple magnetic points. Plastic sheets hung from the ceiling with circular holes cutting them, delineating microenvironments within the larger space. And you can see these wall hangings at the back also of Lee's work. And most curiously an arrangement of Lee's geometric points on a tufted circular rug. Several photographs show Lee encouraging his visitors to interact directly with the work. He also installed a weblight installation with washing lines strung across the trees in the courtyard of Boothby. And one important trace of these exhibitions at Boothby is a tablecloth that served as Lee's visitor's book. And if you examine some of the names there, Audrey Barker, E.J. Hooper, Vineford Nicholls, Hediglson, any one of these names opens up trajectories that are vast. One of those trajectories of course leads to a little known but important exhibition called Popa at MoMA, pioneers of part art or participatory art at the Museum of Modern Art Oxford in 71. Alongside Lee it featured Leisure Clarke, John Duggar, David Madala, Helio Attisica and Graham Stevens. And it reached not a variety for when a jolly young Oxford crowd took the part art manifesto literally. And as Hilary Flo quotes, everything was getting smashed. Two of the artists who were there for the opening, Madala and Duggar, withdrew their work. Lee was happy to let his work be. A show open and closed on the same evening. And it raises questions for the role of participatory art in the museum. And I also closed this section with the proposition that the LYC Museum was Lee showing us how participatory art could live in a museum setting, where the operational activity of running a museum, the openings, the performances, the programming, publishing, could themselves be considered part of an artwork. It suggests that Lee's engagement with place, a close circle of friends and distinctive artistic visions played an important role in shaping Lee's imagination of what the LYC Museum could be. In the LYC, Lee had fashioned a machine capable of changing people's relationship with art, with each other and hence with life. Or, as he earlier phrased it, to make him fall in love with the world again. Now, the second artwork that I want to focus on, I'll start actually with something from the archive. A book produced with Madalyn Huikas. And it documents, it's a photo documentation of Madalyn's day at a Japanese monastery. And the inside cover has this sort of a trace of ownership from Delia Derbyshire with Madalyn's personal inscription to the LYC Library. And Elsin and Madalyn, as many of you would have heard yesterday at the performance in conversation with David Butler, were video pioneers. They met at the Lee's School of Art and Design in 1966, became friends and collaborators. Madalyn worked with Elsa at her film and sound studio, and then they were joined later by Delia Derbyshire. They then made two films together, and when Derbyshire moved to Cumbria, they visited their friend at the LYC, met Lee, ended up showing at the LYC Museum, and becoming Lee's friends. The video point in time dates from when Lee visited them in Holland after the LYC closed. It suggests a journey, both physical and internal. The physical one is dominated by footage of a rocky terrain that seems to be in shock from a moving car and then slowed down. It is frequently abstracted. A modulated drone evokes and then echoes highland winds. We see Lee's hands preparing for and performing calligraphy. An ink stick is pressed against a stone mortar and then ground down through steady rhythmic circular movements. My newt amounts of water are mixed into the ink to modulate the thickness. Too thick and the writing will not be fluent. Too thin and the ink will flow too fast. This is obviously a practice ritual. We follow Lee's hands, dip a brush into the ink, and describe Chinese characters on paper. The movement is not the dramatic flourish of a heroic drip painting, but more the careful listening of the divining stick, or dowsing rod being manipulated over land to locate a source of water. And as the brush touches the paper, we witness the small oceans of ink from which the calligrapher structures meaning. This is a search. A reading that is lent support by the appearance of an enigmatic compass that frames both the beginning and the end of the work. The movement of the needle from true north north to wild gyrations suggests circuitous roots. Point in time encapsulates an exchange of ideas, aesthetics, and effective registers, nurtured through years of discussions with Lee, but also with Delia Darbyshire's electronic compositions. It also hints at a shared interest in Buddhism that inflects both practices, those of Lee and Stansfield Pwydcas. As Dorothy of Frank points out, Buddhists are not primarily believers, but practitioners. A section of Frank's text on Stansfield Pwydcas bears the title, saying what cannot be said, and is an uncanny echo of the title of the survey exhibition of Lee's work, Lee Wan Chyr, tell me what is not yet said, curated by Guy Brett in the year 2000. Both titles emphasise the unknown, the unsaid, and the unsayable, and suggests the primacy of encounter and feeling as a way to experience the practice of both Lee and Stansfield in Pwydcas. Now, if Pwydcas Stansfield were friends Lee knew, Helen Petz was a friend he didn't. Her film, Space and Freedom, was newly commissioned for Speech Acts and takes its name from Lee's search for the same in Cumbria. It begins in darkness.  It heightens our sense of hearing, but despite this preparation, we have to strain to hear a heavily accented male voice whistling and describing a vision for a place. The voice is Lee Wan Chyr, and the place is the LYC Museum. These recordings of Lee's voice were hiding in plain sight on real-to-rail tapes in the still-to-be catalogue LYC archives of the special collections of the John Rylins Library. This is the first time this recording has been digitised, and possibly the first time it has been heard since Lee recorded it. Petz uses this sound recording as one of the central pillars of Space and Freedom. She combines it with previously digitised video shot by Lee on an 8mm film. Films she shot herself on location and original music improvised by Steve Beresford. The film thus becomes a site for collaboration across time and with Lee and Beresford. A collective exploration of the rhythms, textures, sights and sounds of the Cumbrian landscape that's so inspired Lee. Her film has the sensibility of collage and has a companion piece to throw them up and let them sing. Her poetic film on Kurt Schwitter's exile in Norway that also ended up in the Cumbrian countryside. The historian Nicolle liked to work on the dead because they stay still and do not answer back. He said, With the dead there is no rivalry. In the dead there is no change. Plato is never solo. Cervantes is never petulant. The Mostanese never comes unreasonably. Dante never stays too long. But as Petz filmed clearly shows us that the dead can speak to us. But for that we have to be willing to listen. I want to end not so much with a conclusion but a proposition that there is in the discipline of art history an impulse to disentangle entangled histories, to reduce the complex and the messy to tidy pockets of monographic depth neat with sharp edges. I've argued for a need for historical research that exceeds national and institutional frames that is more than singular art historians and individual subjects in single institutions one object at a time. So undertaking inquiries that are cross-disciplinary that cross-disciplinary and geographic borders requires polyphonic, multi-authored and speculative approaches that are described elsewhere as art histories of excess. And in the case of the LYC we here are a part of two ongoing examples of such collective effort. One is speech acts and the second is this symposium which is anchored in the exhibition and brings together scholars, artists, museum directors, curators to consider the idea of exhibitions, publications, place, friendship and the function of museums as a way to think through and think with the LYC museum and art gallery. I'm going to flip through this because hopefully you've all seen this upstairs but I wanted the last word to be leaves. So this is the statement that was in the child children's art room of the LYC museum and we've reproduced upstairs and it reads, everyone can roll. You can write a poem or draw a picture or write a story with a picture put it up on the wall or take it with you or leave it neatly on the table. Enjoy yourself but don't waste paper. The paper is free. Please leave the table tidy. Thank you. So tool we have about five minutes for any burning questions. Okay, tepid questions. It's part of the exhibition permanently it's not just on lunchtime today it's an installation that was temporarily switched off because it wasn't performing itself tonight but you can see it at any of the time because we're going to be at the exhibition. Thank you. Me? Thank you so much for this paper. I think it's methodologically really provocative and an important way of rethinking the way that we write our histories. My question is actually a little bit more banal unfortunately than your wonderful paper. I'm curious to what extent was calligraphy part of Lee's practice because it seems to be quite different from many of the works that we've seen in your exhibition but also in other exhibitions. I'm wondering what role that played in this work. Far from being banal I think that's a great question that I feel not completely equipped to answer if I'm honest. I think there's still considerable work to do and I think one of the things that we might do is hold this question and point it to Leslie Ma a little bit later but there's copious amounts of calligraphy within the archive. He's a poet not just in Chinese but in Italian and English that also says something about his spirit and as you would have seen in Madeleine's wonderful film there is muscle memory watching him perform calligraphy. So it is very much part of I think his vocabulary but how it has been historicised, displayed and understood within part of his wider practice I think remains a question. Charles? How do you write to the music? Was it this group of people that are in cover at the time? Was it local people? Was it people that got the train from London or I guess from London? The short answer is all of the above and Kate, who curates the exhibition with me and has been leading people from all sorts from everywhere through the exhibition I think has been building up a little informal database of personal stories and anecdotes but the sheer number of artists, designers, creative people who experience the museum as a child is sort of fascinating and one of the things that is currently being looked at is the possibility to think of effect that exposure to the museum had long term effect in sort of changing people's life and one of the best stories that Kate shared with me and I forget who it was who said that it was somebody who went to the LYC Museum as a young girl and then described their shock when they found out that not every city has one. And we have within our audience Joy Dee who assisted Lee at the museum and Nick Sawyer of course who brought the two to Cumbria there will be time to table some of these up Joy, please He was known and loved and people went out there from minorities and Norma was very active in those days very very not going on and so all those people were drawn into LYC and it was like we lived in Southern at the time and we would go out there with our kids for a Saturday opening and it was just one of the highlights of the month and of course four new exhibitions together to re-comment no matter what time of year boys over there Sorry No, not at all It's a perfect example of the collaborative work of a symposium Well I think we will have the rest of the day for questions so without that thank you and encourage our first session


Hi everyone, Joe for Jaspy's casebreaks.com coming at you with our last filler That it's gonna unlock NT basketball coming up in the next video. We've got 21 22 prism basketball Hobby pack number 12 three different dice rolls happened here. We got eight spots being given away within the pack Second dice will be the break itself and then the third and final dice roll given away those final four random number block spots All the teams are in when we get to that big. Thanks to this group here We only sold 22 spots outright and we're gonna give away eight spots. So let's do that first Be a little buy one get one scenario for some of you six into two eight times top eight after eight one two three four five six seven and eight So after eight Andy Adams Steven Andy Steven Adam Adam Adam What are extra spots go on your way? So that should give you Not should it will give you better odds to try to win those number block spots Step one done. Let's see if those extra spots will help you out later on. All right now. Let's do the break itself Let's roll it and randomize names and teams five and a six eleven times one two three four Five six seven eight and eleven the final time After eleven we got Steven down to Steven five and a six eleven times for the teams one two Three eleven the final time as you know by now no trade windows in these little filler breaks All right. Here's the first half of the list right over there and The second half of the list right over here. All right now Let's let's print rip. All right. So here's the final print out hot off the presses Thanks for joining me on a Wednesday You release day Good luck. Okay. Got a killing. Hey silver.  That's exciting. It's gather everyone's names in here Let's roll and randomize that list five minutes six eleven times. So top four after eleven Good luck one two three four five six seven eight nine ten and 11th and final time so again can only be the top four. So there's gonna be a lot of sad faces here Especially Andy who's very close But not quite but hey, thanks everyone for giving this shot five to 30 There's another case of NT with another set of filler. So check it out. Jasper's case breaks calm But for pick your team one top four. It's gonna be Stephen flat Carl Stephen flat and Stephen P Congrats the top four after 11 See, this is exactly where you're gonna get Stephen P and Stephen F The Stevens getting a spot each in the Cleveland number block Carl and Stephen flat into that Toronto number block There you go gang. I'm Joe for Jasper's case breaks calm I'll see you for those random number block randomizers and then the break. Bye. Bye


Hello everyone. Welcome to the session. It is more than just correlation at the back journey. So while several Kubernetes users already have access to metrics, logs, to some extent traces, there is still no established open source tool out there that aggregates all these different information and helps users understanding how their systems behave and most importantly why those systems break. So contributing to Soviety's issue is actually the driving motive of today's session at KubeCon. So thank you for joining. But before deep diving into the topic, a few words about us. My name is Vanessa. I am an observability product manager at Redette working in OpenShift. And this is my colleague. Hello, I'm Simon. So I'm working on OpenShift monitoring as an engineer. Great. So let's take a look at the agenda for today. First, we will talk about the problems that engineers, mainly site reliability engineers, have to face when troubleshooting issues in Kubernetes, mainly when troubleshooting clusters. After that, we will take a look at our solution, at least the proposed solution to this problem, Correlator, which is an open source project founded within Redette with the goal of making correlation across observability signals accessible to everyone. After that, we will have a demo led by Simon and then we will recap on the functionalities of Correlator. So stay tuned for that. And lastly, we will wrap up the presentation with a sneak peek overview of our roadmap vision next steps we have around it. So let's get started with section number one. What is the problem that we're actually trying to solve here? Everything started a few months ago. We were wondering how we could facilitate the troubleshooting process of site reliability engineers when dealing with OpenShift clusters. So what we knew back then was that they need to access relevant information to keep the system up and running, securing stable, and avoiding any disruptions to the end customers. And in this troubleshooting journey is where observability signals come into play. Observability signals represent the relevant information site reliability engineers need to troubleshoot. For the ones of you who are not familiar with the term observability, what do we define with it? So observability is about correlating various sources so that you can answer any questions you might have about your running system, assisting you in resolving issues within your system, and also why not optimizing it. So with this information in mind, let's take a look at what we mean with observability signals here. In the Kubernetes world, there are tons of signals, as many of you already know. We have provided you with an overview of the most encountered ones, so alerts, metrics, logs, traces, network events, and Kubernetes events. Alerts are rules that fire when specific thresholds are being crossed. Metrics are numerical values that represent the performance of a system. We have logs that are records of events from both pods and cluster nodes. So we have different types of logs, infrastructure, application, audit logs. We have traces which are end-to-end requests comprised by spans that allow us to view, to track our request a transaction is flowing throughout our system. We have network events which are IP, TCP level network information, and lastly Kubernetes events, which are events that involve specific Kubernetes entities such as nodes, pods, services. In addition to those signals that you see here, I would like to add a couple of additional ones. So we have RAM books that are problem-solving guides, the so-called how-to associated with alerts. We also have Kubernetes objects, which are persistent entities in the Kubernetes system that represent the status of a cluster that includes namespaces, services, pods. So with this information in mind, we have asked ourselves which of those signals were actually used the most for troubleshooting by our satellite engineers. We have conducted a study within Red Hat with a couple of teams and found out the following that you can see from the slide. So there are signals that are used all the time, signals that are used less frequently, and signals that at this point in time are not used at all or more rarely. So the ones that are used all the time are alerts, logs, Kubernetes events, objects, and links, alerts being the natural starting point of many troubleshooting journeys. Metrics are used less frequently for ad hoc investigation, ad hoc querying. And then we have network events and traces, which at this point in time are not used at all or more rarely. But please keep in mind that popularity follows maturity here. Logs are among the oldest signals out there, network events and traces, and instead among the newest ones. So as they may raise in maturity, as they mature with the time, they become well-known, their popularity, relevance, usefulness, and adoption will also increase over time. So this overview is likely to change. But we still use it as our starting point for our correlators. So bear with us. We have also asked ourselves how does a typical satellite-based engineering workflow look like. Here you find two examples. So we have an alert being the starting point of the investigations alert via pager duty via slack notification. And then in gray you find every step needed until the problem is resolved. If we take the workflow number one, we have, for instance, an alert as a starting point, the SRE would go to the relevant SOP or RAM book, would go to the relevant application dashboard, looking into the relevant application pod logs at all their logs whenever needed, looking also at pod events or probes until the problem is restarted and is actually resolved. And in this case, the pods restarted. The second workflow, for instance, we have still an alert as a starting point, but then based on the SRE preferences, we go to check the cluster health indicators. So the base ones are all other alerts on the cluster as well. And then not in a specific order, the relevant logs, metrics, built-in dashboards, or Kubernetes events, and if needed also the cloud objects status is also being checked. So we do not want to focus on the specific steps here of this workflow, but the main learning out of this slide is definitely troubleshooting means jumping from one observability signal to another. As you can see, the variety of steps that we have highlighted in gray. Here you also see the time spent on individual clusters by one of our team in Red Hat. So each activity is represented by a circle. The bigger the circle, the greater the time roughly spent on individual clusters. So as you can see from the slide, we have the circling green, which represents the hands-on troubleshooting and factor gathering. So that requires the greater amount of time. This task is also complemented by other activities such as query metrics, reviewing collecting pod logs, reviewing audit data, but also reviewing cloud provider config and audit data. So a lot of time spent on individual clusters here. And this means also a high cognitive load for satellite engineering when troubleshooting. So to summarize, what are the day-to-day challenges here that we have seen? It takes time, especially for newcomers in enterprises, to gain enough familiarity with all the systems up and running, but also with the complexity of Kubernetes. It takes time and effort also to learn how to query the relevant information, the factor gathering part. But not only for newcomers, also for more seasoned experienced colleagues, metadata and API heterogeneity makes it hard to find the possible root cause of the system problems. So we have asked ourselves one question. How can we provide satellite engineers with the observability signals they need, most importantly when they need them? And this leads me to the next section of the presentation, the actual solution, or at least our proposed solution to this issue, Correlator, which is an open effort to fill this gap. So we have seen from the previous section that reducing, minimizing the time spent and so on on the cluster and allow SREs to identify and focus on the real issues that need to be investigated is fundamental for us. And that's where correlation across observability signals becomes an asset. What do we mean here with correlation? Correlating observability signals means following relationships to find related data in multiple heterogeneous stores. So once again, for clarification purpose, we refer to the broader connotation of correlation, meaning an association between two and more variables, not the statistical connotation of a correlation. So to wrap it up, why we think that correlating observability signals is an asset is so beneficial because by bringing together observability signals, we want to allow engineers in this case, satellite ability engineers, which were our first target persona to focus on the diagnostic data that matters at the end of the day. So eliminating the manual efforts that we saw in the previous slides in the gray steps of the workflow, so of navigating from a signal to another. So we want to reduce the time spent by SREs troubleshooting. Thank you, Vanessa. So I'm going now to show you a little bit more about correlator, how it works in practice, what is the design, the architecture. But first of all, I need to stress that this is a very new project, very young. We started that a few months ago with really the goal of helping our internal SREs who manage lots of OpenShift clusters. So what you will see in the demo is using OpenShift really eventually or even in the short term, we really want that to be applied not only to OpenShift clusters, but really all type of Kubernetes clusters. The main goal really of correlator, like Vanessa said, is to help you navigate and gather facts about related information. And really building relationships between different entities. So like we are going to see going from logs to metrics to alerts and also Kubernetes objects. The tool itself is written in Go. We are like our only target is Kubernetes clusters. So that was a natural choice. We envision like right now it's a binary that you can run as a CLI as a web UI. But we also think that it can be useful as a library but we again, we wait for more use cases before we dive into this. The overall architecture is quite simple. On the left side, you see we have a user interface that I will show in the demo. Again, keep in mind we are very early stages. We are not UI developers. Obviously, we will see that. We have also a CLI. The API here is something that we need to design and to think through. But that's also in the plan. On the right side, you will see all the different stores, as we call them, that we can integrate with. Obviously, Kubernetes API and lots of like for metrics, we support anything that can talk to the Prometheus API. For the logs, we support Loki right now. Because again, this is what we have in our OpenShift product. But these are solutions that are not limited to OpenShift. If you do monitoring on OpenShift, then you probably use Prometheus, or Thanos, or even Mimir. Everything that can speak to the Prometheus API is fine for us. So you will see the different signals. Again, when we say signals, it's not like only observatory signals, you will see that we kind of, I would use signals also for talking about Kubernetes objects resources. At the bottom, this is really the core of correlator, are the rules, which are where we are going to draw relationships between the different data stores. And now probably the best option is to go to the demo so that you can probably see a little bit more what we are talking about. So on the left side, I've got the correlator UI. As you can see, very bare bone. That's the current state. On the right side, this is the OpenShift console. So again, if you are not familiar, that's not really important. What is important to see there is I've got an, I'm on an alert page. So what I'm going to do is I'm going to ask correlator, okay, what can you tell me about that alert? And so I'm going to put the address of that page into the correlator. And I'm going to ask it, okay, can you tell me about the logs? And it's going to query the different data stores and going to draw that graph. So I can see that from the alert, I can go up to the deployment here. From the deployment, I can go to the events, Kubernetes events, I can go to the pod. And finally, what I was asking for are, okay, show me the logs. And I can click on that age, which will show me basically jump me to the console page where I can see the logs of my deployment in that case. And I can quickly see that, okay, my deployment or my alert is firing because my deployment is crash-looking. So that's the basic scenario. I know where I want to go start from an alert. Let's take a different alert now. I've got, okay, I just want to pick the good one. I've got another alert firing, which is kube container waiting, basically telling me that I've got a container which can't be started. I'm going to do the same. So take the, again, this is a shortcut or an easy way for me to ask or later show me this is the starting point. I'm going to again ask it, okay, can you show me logs for that? No, no match. You can see I've got, again, my alerts there, but nothing on the log side. Okay, maybe I should look for events then here. And this is going to give me a different answer. In that case, again, I can go from the alert to the pod and to the pod to the event or events associated to that pod. And here now it becomes obvious why my container doesn't start. It's because I've got a very bad image location. Now these two scenarios are quite simple. It's I know where I want to go. But what if I start from instead of starting from an alert, let's say that I start from a deployment. Okay, let me take that one, which is still in a bad state. And I'm going to ask correlator again, can you show me all the objects that relate to that one? All the signals. And this is going to draw me a map of something which is going to be more complete or more exhaustive. So again, I'm still starting from the green circle is my starting point. And now I see that I can, I've got a full picture of all the events, all the signals related to that Kubernetes object. Again, there are some links that are quite natural going from the deployment to the bodies. Easy, you should, you might say. But again, you see that we can, for instance, we can ask it or show all the metrics related to that part. Again, this is the current state. You can imagine many, many different like having additional links, additional objects that you can draw there. And finally, to close the demo part, this is like, this is showing you all the potential path starting from a deployment where you could go from different signals and different objects in every signal. So we can see that from a deployment, we can go to the event, we can go to even Kubernetes ingress, et cetera, et cetera, from logs, and so on. So now that we've seen like the basics of correlator, maybe a little bit more about how it works in practice. Again, the, the meat of correlator are, are what we call rules, where you would basically rules is a way to say, given the starting point, like in this example, the alert, I want to go from the alert to my Kubernetes deployment. So this is the goal parameter here in the file in this YAML file. And the way the rule will infer the going from the starting point to the goal point is using, is basically having a query, which is like a correlator query in a way. The query itself, so as for those who are familiar with Go, this is Go template right now. That was the easiest way for us to model that, but we are not sold to it. So the, the query itself will get as an input, the starting objects. So in that case, the alert, and you see here that it's going to expand the labels of the alert. So any alert that will have a namespace level and a deployment level will match that rule. And basically that will tell correlator, okay, show me all the deployment or show me the deployment in Kubernetes that has this namespace alert level as a namespace and same for the name. So this is how we can go from one object in a given domain in a given class of signal to another object. So this is again just to clarify things domain are right now what we have as domains are Kubernetes for all Kubernetes resources. So obviously in each class in that case would be a pod, a route, an ingress, a deployment, an event, et cetera. So you can have many class inside the domain. We have also another, like we have the alert domain that we've seen. We have a metric domain, we have a log domain, et cetera. The stores on the right are the actual concrete data stores, the APIs. And the role of the domain in the correlator project is to translate like the internal correlator data model and query format into something that maps the Kubernetes API or the primitive API or the low key API. That's basically the domain is the translation from this expert knowledge into something concrete that you can query against a data store. Just to show you again, so let's say that I want to go from that green class to that red class on the right, then correlator will start from that green class, evaluate all the rules whenever there is one matching or a query that matches, that returns some results, then correlator will draw an edge from that node to that other node and will repeat the same operation again. So for the intermediate class, it will again evaluate all the rules that it has knowledge about. And finally, you can see that we end up with a direct acyclic graph where the nodes are the class or the objects and the edge are the rules. So why we found correlator to be interesting and to be a bit different from what exists today? I think the first reason is really because we are integrating Kubernetes knowledge into that fact gathering. You can troubleshoot a lot with logs and metrics, but sometimes you miss the Kubernetes context. Are there other events related to that object? What's the status even the status field in my resources might be useful? Even the levels or the annotations, all the things that can be really represented as metrics. This is for us what makes it different compared to what we have now. The other thing is really being able to codify or encode the SRE knowledge or the operation knowledge into something that can be automated, which is again, you can have like Vanessa said, you can have rent books, but that means that people have to do the operation or to do the query themselves. Finally, the third point is really we want that to be flexible. We want people to write their own rules or to submit us with new workflows that we didn't anticipate. This is really the idea is that you can extend that as you wish. Again, to summarize, our main focus right now is really to speed up the debugging journey so that you don't spend instead of spending 15 minutes gathering all the information which is spread into different data stores so that you can have that at a glance. I think now it's your turn. Thank you, Simon. So now time to wrap up the presentation with an overview of our roadmap. So what's our goal here? So we want to provide a correlation engine that is flexible enough to be embedded into graphical user interfaces, but also into the command line interfaces, as well as provide it as an offline tool for data processing tasks, as well as an API to answer different user needs and preferences. So who do we want to support? We have seen the SRE journey, troubleshooting journey, but we are aware that there are so many others, including IT ops, the developers, persona, so that includes software engineers. So not necessarily starting with an alert as a starting point for investigations. So we want to answer different needs here. Just as a quick reminder that we just focus on the SRE persona as our starting point because that was our biggest use case in Red Hat in OpenShift in this case. So what does our roadmap look like for the near-term, mid-term and long-term? So definitely we want to enhance our set of rules that Simon has also shown in the demo. So not only looking into alerts through logs, but of course other possible scenarios here. Also focusing beyond OpenShift, for the ones of you that are not aware, OpenShift container platform is Red Hat's distribution of Kubernetes. So here we want to support all other Kubernetes environments. We also want to improve the graphical user interface that Simon has shown in the demo. So also stay tuned for that. And in the mid-term, we also want to improve and provide offline services experience, as I showed before in the previous slide, and improve our command-line interface experience. And lastly, in the long-term, we want to simplify how we capture the domain knowledge through the rules that we have seen before with the anatomy of the rule that Simon has explained. So are you interested in learning more about Correlator and our community? You can engage with us. We have a GitHub repository. The links are provided in the slides. And also do you work yourself with Kubernetes clusters? You can provide us with a feedback on your use cases so that it can also develop our additional rules. So you can file a request for enhancement. We have a template. So feel free to do so. That will help us enhance the journey as we are at the starting point. But it's pretty much exciting. So that's about it. So do you have any questions? We will answer the questions on the virtual platform after the session. But in the meantime, if any of you has some questions for us, feel free to let us know. Thank you so much.