The Congress Party has been routed in Gujarat and it might have won in Machal Pradesh. But the Congress hasn't won a single big state or high stakes election since the winter of 2018. That's four long years. Now why can't the Congress win any big election? Mainstream media will tell you it's because Rahul Gandhi is a failed leader. He cannot win votes for the Congress. He's so unpopular that he might as well end up losing votes for the Congress. Mainstream news anchors will also tell you that the Congress just doesn't work hard. Look at the app, they'll say, out there battling for every last vote. And the Congress, you can't find them on the ground. And Rahul Gandhi is himself like that, absent every time he is needed. If you watch television coverage on Counting Day on India's main news channel, its national channels, you'd have seen how every anchor, every analyst said, how can the Congress expect to win elections when Rahul doesn't even campaign for his own candidates. He was busy with his Bharat Jodo Yatra at the time when there were these crucial state elections taking place. Now these points seem to be absolutely valid, but in reality they're misleading. And I would say they're deliberately meant to mislead you, to divert from the real reasons why the Congress is not able to put up a fight. And these are five key reasons. Three M's, media, misinformation and money power. And there are two P's, political ideology and polarization. I'll start with the media. When was the last time you saw any positive or even neutral coverage of Rahul Gandhi or the Congress party in mainstream media? Rahul is walking from Kanyakumari at the southernmost tip of India to Srinagar, a journey of 3,750 kilometers on foot. Have you seen any detailed coverage of this on any TV channel? It only hits the headlines where Rahul says something, which mainstream anchors can then spin into a controversy and then attack him. This is how it has been since 2014 and even earlier when a section of the media turned against Rahul Gandhi. For the past 10 years or more, the Congress party has only got bad press and it has been done in a calculated manner, often orchestrated by top journalists and editors. This is mainstream media. On social media, it is much worse and that brings me to point number two, misinformation. There is deliberate fake news being pushed out against the Congress party and its leadership and an entire political machinery is behind it. False stories, fake information, courts taken out of context, edited videos. They're all spread via WhatsApp, Twitter and Facebook and then they're amplified by influencers and politicians from non-Congress parties and then once lies have been repeated many times and drilled into the minds of the public, they return to mainstream media as facts. And then mainstream media gets into the act to push it out even more and they still have the audacity to claim that the Congress isn't putting up a fight when they're deliberately participation in diminishing the Congress party trying to defeat it. But why would the media become an extension of the government and the ruling party? This is where money comes in, money power. Big media is now controlled entirely by big business and a few of those who aren't are entirely dependent on advertising for the survival. So they do what corporate India wants them to do. They say what corporate India wants them to say if a journalist has an independent mindset or believes that journalism is about questioning those in power, whether political, institutional or financial, such journalists have been sacked and sidelined in the past five to six years. Those are willing to toe the government's line, do what their corporate masters tell them to, they're promoted, they get prime time shows, they become editors and get visibility. That is how corporate India's message, its ideologies push through these media stars to the average citizen who then accepts it as authentic news as facts. But he will ask why is corporate India against the Congress, especially Rahul Gandhi and that brings us to Congress party's political ideology. Actually Rahul Gandhi's political ideology. He has openly positioned himself as anti-big business and he did it way back in 2010 when he went and batted for tribal rights in Orissa against a Vedanta mining project which then got scrapped. That was a warning signal for India's big business that this man could be a problem. He might try to take India back towards a socialist path where the state controlled the economy. Since then Rahul has only strengthened that view. Everything he has said has been about equality, about the poor and he has also surrounded himself with left-leaning advisers, at least that's the allegation against it. And after the Congress was defeated in 2014, Rahul's attacks on big business houses have only increased, they haven't reduced. So the richest business families and companies in India have every reason to want to keep him out of power and to finance anyone who can keep Rahul out of power. The problem for corporate India and its compliant media is that Rahul's politics could actually well work in a country where 70% of people are poor where the top 10% own most of the wealth and corner most of the country's income. The best way to stop the poor from uniting on economic issues is to make them unite under the banner of religion, in other words, to polarize them. And that is what is increasingly happening in India. Where the national media, social media, those on power, those leading institutions have all united to push a polarizing ideology. Hindus versus Muslims, majority versus minority, the entire ruling classes are united to attack secularism and liberalism which makes it very difficult for the Congress Party to promote its own politics. It is in this atmosphere that the Congress has to fight elections with odds heavily tilted against it. When the field is so badly skewed against the party, it is very difficult for the average party worker or even local leaders to feel infused to push themselves because they feel the battle is lost even before it begins. And the mainstream media pretends that it has no role in this, that it is entirely the Congress Party's own fault that it cannot win elections. The reality is very different. If you have liked this video, do press the like button. It will help YouTube's algorithm recognize that you liked it and it will then reach more people like you. And do subscribe to our channel. You will get to see what corporate controlled mainstream media will never show you. Until next time, goodbye.


All right, can everyone hear me okay? Yes. All right, I guess it's time to start. My name is Sage Weil. I'm the Ceph Principal Architect at Red Hat, and today I'm gonna talk about Ceph, Manila, and a little bit of Back Containers in OpenStack. So just a brief outline of my talk. I'm gonna give a bit of background on Ceph and CephFS. I'm gonna give an update on the current state of CephFS. Gonna talk about what the current landscape in Manila looks like. CephFS native driver we're working on, and then sort of segue into our plan for better file system plumbing into virtual machines in OpenStack, how that affects Manila versus Nova, and how the responsibilities might break down, and what that means for containers. So let's start with Ceph and CephFS, or actually why should we use files in the cloud instead of say object storage? The biggest reason is that file-based applications aren't going away, POSIX is lingua franca in the computing world. It's important to be able to interoperate with other storage systems and data sets. As more people start using containers, container volumes are really just directories in a file system, we probably wanna be able to support that effectively. And it turns out that permissions and directories are actually useful concepts, so we shouldn't just throw them out because they're something else. One might ask why not just take a block device and put a local file system on top? And the problem there is that block devices really aren't shared when you put a local file system on it, it expects exclusive access to that block device, so it's not useful for sharing data between virtual machines. And also block devices aren't very elastic, it's difficult to expand and contract the file systems there, whereas shared distributed file systems usually do this quite naturally. Why should we look at Ceph? Ceph is designed from the ground up to scale horizontally, which makes it a very good fit for the cloud. It has no single point of failure. It's hardware agnostic, it's designed to run on commodity hardware. And because it's designed to run at scale, it's self-managing whenever possible. And importantly, it's also open source, completely free and open source, which makes it a good fit for open stack. Also, as we're building Ceph, our goal has always been to move beyond legacy approaches. That means that we try to adopt a client cluster paradigm where the clients that are talking to Ceph understand that they're actually a whole mess of servers that are storing their data that might be failing and migrating things around and they're fully capable and prepared to deal with that situation and start talking to the right server in order to find the data that they need. So this no Ceph talk would be complete without this diagram, high-level Ceph architecture. It's all based on RATOS, this red bit at the bottom. It's a distributed object store that manages the distribution replication of all data in the system. If nodes fail, then it migrates data away from them, re-replicates it, makes sure that your data is separated across failure domains and those cluster expansion contraction, all that good stuff. And then on top of RATOS, we build a number of different high-level services. There's the RATOS gateway, which gives you S3 and Swift-compatible API object storage. There's RBD, which gives you block storage, which many of you I'm sure are familiar with. And there's also CephFS, which gives you a fully distributed POSIX file system that sits on top of RATOS. And it's this last bit that I'm gonna talk about today. So CephFS stores all of its files as objects in RATOS that the clients access directly, which means you have very scalable and high-performance access to your data. It scales just as RATOS does. It also has scalable metadata access because the metadata in the system, the files and directories and permissions and so forth are distributed across a whole set of metadata servers that are dynamically managing that file system namespace. And it provides you a POSIX interface, so it's a drop-in replacement for any local or network file system. There are multiple clients that are implemented for CephFS. There's one in the Linux kernel that's been there for many years now. There's a user space implementation that you can access via Fuse. That's called Cephuse. And you can also access it via libcephfs.so, which is a shared library that can be linked into Samba to re-export the sys, Ganesha to re-export via NFS. And it can also be linked into a dupe to run big data type analytics workloads. The thing that truly makes CephFS unique is its use of dynamic subtree partitioning. And the idea here is that you have a whole hierarchy of files in your file system and CephFS will sort of on the fly take pieces of that hierarchy and carve it up and distribute it among the different metadata servers in the system. Can do this pretty arbitrarily. I can even take a single directory that has lots of files or is very busy and shard that into lots of little pieces and then distribute those fragments across different metadata servers. This makes the CephFS metadata server cluster highly scalable because we can arbitrarily partition the hierarchy into little pieces, supporting tens or hundreds of metadata servers. It's also adaptive because this partition is based on the current workload. So as different jobs start up and users start using the file system, then the metadata servers will migrate load from the busy servers to the out of servers and adjust that partition over time to fully distribute and take advantage of all the servers in the cluster. It can even take hot metadata and replicate it across multiple nodes so that you get the best performance that you can. There are a lot of other good cool things in CephFS. It has strongly consistent and coherent client caches, something that NFS has never had which means that it's gonna behave the same way as a local file system will. It has a recursive accounting feature that's relatively, I think, completely unique actually and that it keeps track hierarchically of everything that's stored in the system. So if you look at the size of a directory instead of getting sort of a bogus 4k number, I mean it'll tell you the number of bytes that are stored in that entire subtree of the hierarchy. Same thing that you would get out of a DU for free which is pretty nice. You can take snapshots on any directory in the file system. You have directory quotas where you can limit the size of a subtree based on the number of bytes stored or file stored that supports X adders, ACLs on the kernel client. There's also support for client side persistent caching using the kernels FS cache subsystem. So, lots of good stuff and we've been putting a lot of work into it. CephFS is actually where Ceph began some 10 years ago and sadly it's also sort of the last thing to become production ready but we're changing that as we focus over the last year primarily on resilience. So our focus has been on handling errors gracefully, detecting reporting issues that we see in the system and providing recovery tools in case things go horribly wrong. And our goal is to really achieve this first in a single metadata server situation so that we have the highest confidence that things are gonna work and you're not gonna lose any data. Part of this has been dog booting this system in our own environment. So the Ceph QA infrastructure is relatively large and storing lots of data and so forth. We've been running all that on CephFS for more than a year now. And we found it thick several sort of hard to reproduce bugs mostly in the kernel client but also a few in the MDS. Things that you would only really see running in a production environment but we're happy to say that it's been quite stable for some time now, at least on the CephFS side we've had other issues because it's running on a really, really old terrible hardware but those things aside we're quite happy with it. Which makes me happy to say that we plan to call CephFS production ready in the next stable release jewel coming out in the quarter one of this coming year. So that's very exciting and it's been a long time coming. Yay. Yeah. Yeah. So as I said there's a lot of work that's been going into CephFS. That includes improved health checks for diagnosing problems. You can identify which clients are misbehaving and which OSDs are misbehaving. There are lots of diagnostic tools so you can debug the system, figure out what requests are passing through the MDS if they're hung or something like that. There's better full space management. There's client management for evicting those misbehaving clients. And also work on continuous verification so that you can scrub an online system and make sure that there are no errors without taking it offline and doing some sort of consistency check. But our real focus again has been on this FS check and repair because although there are no real known issues with CephFS that we've been trying to fix it's not that stability has been bad. It's really that we wanna make sure that we have the tools available to recover data. If we do have people in the wild running it sort of internist that run into problems we wanna have some confidence that we'll be able to fix their problem without having to write custom tools to go do that. So we want repair tools that handle the loss of data objects in case RATOS totally fails and loses some of your data. And then also more importantly, tools that will recover the loss of metadata objects in case the metadata for the file system gets corrupted we'll be able to rebuild that. So there's a CephFS journal tool that we put together that's sort of a disaster recovery for the MDS journal structure that can pull recently written metadata out and put it back into the metadata pool. There's a table tool that adjusts some of the other internal metadata structures. And the last bit is probably the most important is the CephFS data scan tool which is can be used if there's a complete loss of all the metadata in the system you can actually rebuild the file system hierarchy from all the metadata that's attached to the file objects in a separate RATOS pool. So in case things go horribly wrong we'll be able to get your data back. And all that is gonna be ready for Jool so that's very exciting. One of the other new things is around access control. So we had a couple of interns over the summer the first was a Google Summer of Code student who was working on path-based authentication. The idea here is just to take a client and restrict them to a sub-directory mount within the system. That's now implemented in the MDS. There's one missing piece that will also allow them to be locked inside a RATOS namespace that will be in the Jool release. So thanks to Josh on from Google Summer of Code for that. We're also working on a user-based authentication system which is slightly different. The idea here is that you have a client that's mounting as a specific UID and GID and then we enforce some UNIX permissions on the server. That eventually we like to also wire into sort of external off-off frameworks like Kerberos or Active Directory to be used in sort of multi-user, I guess, IT environments. And that was an outreach-y intern student. So that's all new. So that's what's new in CFFS. Lots of excitement there which brings me to OpenSAC. So I'm gonna start a bit by talking about the current landscape in Manila and then we'll see how CFFS sort of fits into that. So Manila, as you all know, manages file volumes or file shares. You can create and delete shares. It manages some of the file server network connectivity to the tenants. I'm gonna have some stuff around snap stock management for those volumes. But there are also some awkward bits with the current sort of user experience for Manila. Manila only manages part of the connectivity problem. So it creates these neutron share networks that expose the file shares to a neutron network but it's sort of the user's responsibility to then attach the tenant to that network and actually go and do them out on the system. So this sort of last mile is up to the user. So Manila's only sort of managing half of that problem. It's also sort of assuming that the way that the files are being attached to the tenant is network-based. So it assumes that you're using something like NFS or CFS or CFFS that is the network-based protocol which isn't always necessarily what you want when you start thinking about containers. So more about that a little bit later. So most of the Manila drivers are appliance drivers sort of in the open stock tradition. The idea here is basically that you want to be able to tell an appliance to create an NFS share and export it to a particular IP and then create the neutron network to actually do that network plumbing. There are lots of these drivers from the usual suspects. Now the only thing real thing worth pointing out is that the security in this model because the tenant has access to the file server over that neutron network then the security is sort of punted to the appliance. It's assumed that the appliance is effective and secure at actually enforcing the security restrictions that you have that you can do that which is generally true. There's also a generic share driver that is not proprietary, it's fully open source. The idea here is to take existing open stock components and build a file service out of that. So you start with a Cinder volume and you attach it to a service VM that you create put a local file system on top like XFS, run a Ganesha NFS server there and then share that over neutron to your tenant. The nice thing is that we're building out of existing components and it's all fully open and usable and it gives you that tenant isolation so the tenant can't talk to the storage network where every actual storage is coming from. So it gives you that security. On the flip side, it's not the most efficient thing. You have this extra network hop, you have to go through the service VM in order to reach your storage so it's not gonna perform as well as it could. You have this extra VM that you have to run that consumes additional resources and that VM is also a single point of failure if that note goes down then you lose your file service even if your backend storage system is highly available and expensive and so forth. But this is a reference driver, it's in Manila, you can use it today. So that's all good. There's a very similar set of drivers that are built around Ganesha as well and service VMs. So currently this is used for GlusterFest but you could do the same thing with CefFS. So the idea here is that you take this Ganesha driver toolkit that they built, you start up a service VM, that's again running Ganesha but instead of using a Cinder volume, you use the Ganesha SL sort of backend abstraction and have that talk directly to the backend distributed file system. And then you export NFS to the tenant. But again this has, it's nice that it's sort of a simple existing model that we've already used with the reference driver and it has this good security where you're isolating the tenant from the network where you're distributed file system back into section running. But again the problem is that you have this extra hop, the service VM is a single point of failure and consumes additional resources. So this Manila Ganesha toolkit is written sort of modularly, it's used by the current GlusterFest share driver in Manila. It's not yet something that we've implemented in CefFS for reasons that I will talk about right now. So in our view that the problem with the service VM approach is that the architecture is somewhat limited. It's always going to be slower because you have this extra hop through the service VM and it's always going to be expensive because you're running the sex for VM doing all this extra work. The other problem is that the current implementation isn't highly available. So if you need some service monitoring to make sure that VM doesn't go down and if it does you have to restart it. It's a single VM, there's no sort of HA or low distribution or anything like that. And sort of an implementation detail. The way the current Manila code is written it's sort of assuming that there's a single service endpoint that's doing this process. So a lot of that's fixable. There's sort of a big to do list to make this sort of something that's fully robust and usable. But before we invest all that effort to do that I think the question that we need to ask is is this really the right endpoint? Is this what we really want to plumb access to tenants to Cef I guess in our case? Is that really what we want to do? And this caused us, I'll get to that in a moment. So the starting point here is we first built sort of a simpler approach, avoid the service VM and let's just give the tenants the ability to mount CefFS directly. So we made a CefFS native driver as a reference. So the idea here is that you have the tenant, you just give it access to the storage network so you ignore the security implications for now and you mount CefFS directly from the tenant VM. So the nice thing here is that you get really good performance. You have a direct network access. The client talks directly to Cef. You have access to the full CefFS feature set. Use the native kernel driver and it's nice and simple. But there are a few drawbacks. The guest has to run a modern Linux distribution that actually has an up-to-date Cef client. It exposes the tenant to the Cef cluster network which might be a security concern in your environment. The networking is currently up to the user so it's up to the user to make sure that that tenant has access to the storage network. And sort of a detail, the tenant client needs to be delivered a secret that it uses to mount the file system. Although I guess in the Manila session today, they sorted that out so that's coming soon. One of the things that made this driver simpler to implement is to implement first a library that we use to manage the CefFS volumes or shares that we're providing to Manila. So this is a CefFS volume manager.py that wraps, sits on top of the libceffs Python bindings which then talk directly to Cef. That's gonna be packaged as part of the Cef Python RBM and DEB. And the basic idea here is that Manila shares or volumes or just directories in CefFS. So you might have slash Manila, the consistency group and then the volume ID. And this library just captures all of the basic volume management tasks that you would expect. So you can create volumes, you can delete them, create snapshots on a volume or on an entire consistency group which is really trivial in CefFS. You just have to do a maker inside the sort of magical snapshot directory. And then also one of the other API calls in Manila is the ability to take a snapshot and promote it to a volume that you can then mount. I think that's the only way you can actually access a Manila share. And so if you want to read write then you actually have to make a copy because CefFS doesn't support writable snapshots. Probably never will. Or if you want to read only one you can just create a sim link. But the nice thing is that because the volume manager sort of wraps up all this complexity and is packaged as part of Cef then the Manila driver is really simple. It's only 250 lines of code. Which is nice. Excuse me. So the challenge here is that is really the security. So the core issue is that the tenant now has staff access to the storage network so it can talk directly to CefFS. Which means that Cef is the one that's responsible for enforcing whatever security restrictions that you have. So Cef actually has had proper authentication for years. Since like 2009. And our authentication model is modeled after Kerberos which provides the usual authentication of client and server. The client knows it's talking to the right server and the server knows that the client has the secret and is the right one. So what's really new here is that we've added this additional authorization that allows you to say that this particular client if they authenticate is only allowed to access a particular directory in CefFS. And that's the Google Summer Code project I mentioned is now extreme. And there's this one missing piece around Ravus namespaces that again will be present in Jule that I release next year. So the real question then is that enough? Is that a sufficient security model to give you confidence that you're gonna give tenants access to your storage network? It means that Cef security is the only barrier. Cef hasn't been hardened against denial of service the same way that many other systems have. That may or may not be a problem in your environment. I think it really depends as whether this is sort of a public cloud where your users are totally entrusted or whether it's a more trusted environment like a private cloud. And it's something that you as an administrator has to answer in your context. But the security is sort of, it's a bit of a problem. So what we're really looking for is a better way to plumb file access to virtual machines so that we can have something that we can use in a totally entrusted environment like a public cloud. Which brings me to this discussion. And so we run a couple of things. We want better security. We want the same sort of protection and isolation that we have with block storage. We want simplicity of configuration and deployment. Just as we do with Kinew and LiveRBD. The host just attaches the block and it's there for the guests and you don't really have to do anything special. And we'd also like good performance. Ideally, performance that's similar to what we would get with the native access to CephFS. So there are a couple of different models that we've looked at to try to achieve this. The first one that we looked at that initially we were very excited about was the use of 9P in KVM. So 9P is a file sharing protocol analogous to NFS. And Vertifes is an embedded 9P server that's part of Kinew. So the idea is that Kinew itself has a file server that exports just to the guest. And then the guest kernel mounts over 9P using a VertiO transport to make it nice and fast and efficient. So the idea here is to link LibsFFS into Kinew the same way that LiveRBD is. And it sort of links directly into this Vertifes embedded on 9P server. And then the guest OS, which has to be Linux in this case, uses the special version of the 9P protocol that IBM updated like five or seven years ago to mount Vertifes, which is actually re-exporting LibsFFS over the network. So a very similar model to the way that LiveRBD is used today. It's sort of easy to deploy. The nice thing is that this gives you good security, the tenant is isolated again from the network, from the storage network and locked inside a particular directory. And it's really easy to deploy. It's sort of frictionless in the same way that LiveRBD is. But on the other hand, it requires modern Linux guests that have support for 9P. More importantly, 9P isn't supported in most Linux distributions today, at least not the Red Hat ones. I found out after talking to all the developers that there are actually two different reasons for this. One is that the 9P kernel client, the kernel developers don't really like and don't really want to support. And on the Kimi side, the Vertifes server, the Kimi developers also don't really like it. They have concerns about the code quality. So on both fronts, it's not the most attractive option. Now to be fair, code quality is something that can be fixed. It's open source code. That's the whole point. So that could improve over time. But the other issue is that 9P just isn't the best file sharing protocol, at least in my opinion. It's second to NFS or something like native stuff of us. But it's an option. There's in fact a prototype put together by some developers at United Stack. That includes both the Kimi patches to plumb live stuff of us in there and the Minola driver that orchestrates FFS and some changes in Nova to actually do the attachment through LiveBird and so on. But we really like something better. So why don't we look at a different file system protocol that we all know and love. We can just use NFS. So the idea here is that you would mount stuff of us on the host or maybe use Ganesha and then re-export NFS to the guest OS over a private network that just the host and the guest share. So again, you have the same isolation of the tenant from the storage network, which is nice. NFS is supported by everyone, which is good. It's reliable in the sense that the NFS server gateways in the same fault domain as the guest. So if that host goes down, everything fails together so it doesn't really matter. And it works for any file system, not just FFS. On the flip side, NFS is kind of weak cache consistency. It's a better file sharing protocol, it's not the best one. And the protocol translation will slow us down a little bit. But the main issue here, I mean, those are probably not the big of an issue, but the main issue is that the sort of awkward and insecure networking to have this sort of simple private network that's attaching the host and the guest for a couple of different reasons. It means that you have to add a dedicated network to the virtual machine. You have to configure a local subnet on the host and assign MPs to the host and the guest. You have to configure NFS in the hypervisor and you have to then mount that to the virtual machine. The community guys tell me that it's awkward to have these special purpose network interfaces that you attach to the KVM instance that are sort of external to what the user is asking for. I don't know enough to actually know why that is. It also means that you're sort of dependent on the networking configuration inside the guest. So things like firewall D running inside the virtual machine might break this. If the user just restarts networking, their file service might go away, which could be problematic. And also we'll just see that there's this weird network interface and network configured that they might not realize is actually important for getting their file access to the root file system, which can be problematic. But I think that one of the things that concerns me the most is really the security implication where other services on the host might inadvertently be exposed to the guest. So for example, you might have a demon like SSH that's binding to all IPs on the host system that the guest can then access over this private network. So hopefully this admin has far well set up to prevent that, but we know who knows what's really gonna happen. So we'd like a better alternative. And the one we've found is called Vsoc. So Vsoc is actually based on VMware Vsockets. It's a newest address type family. That's designed specifically for communication between virtual machines, hosts. Excuse me. So it's stream-based or connectionless, just like IP. This address is like dead simple. It's just an integer to identify which virtual machine you're talking to and one is the host. And it's been supported in Linux since version 3.9. So we're over two years now. And then the key attraction here is that zero configuration. So the hypervisor always has address one and the VMs are just assigned addresses above that, but nothing has to be configured inside the virtual machine. Once you start up Kimiw and tell it which Vsoc address it is, it has it. So it's all there. So the approach to the idea then is to do this NFS export from the host of the guest over Vsoc instead. So there's some details here. We have to use NFS V4.1 only. That's because older NFS versions have all this annoying protocol stuff that requires addressing. So older versions of NFS use block D for example and that address gets shared and it's a whole mess so you can't really tunnel it over Vsoc. Whereas V4.1 consolidates everything over a single connection, so that's good. It's also much easier to support because it's easy to add support for Vsoc to existing services. It's really mostly boilerplate code to support a new address type and then adding parsing for the way that you actually parse the address. So there are patches in flight for the NFS client and NFS server from Stefan, who's a Kimi developer. There are patches for Ganesha to add Vsoc support from Matt Benjamin and there's also patches to NFS util so that the mount command line utility will work. Make this all work. So this is all sort of shiny, brand new, but we have prototypes working and running and it seems very attractive. So the overall picture then is that you would mount stuff of us on the host and re-export it via KNFSD to a particular Vsoc address for the virtual machine or you can use Ganesha to do the same thing if you don't want to use the Chrome client. And then you would export it just to that virtual machine's Vsoc address. So the nice thing is that NFS is well supported, even NFS 4.1. The security is better, it's simpler configuration, it's more reliable. The only real drive back to this approach from our perspective is just that Vsoc support is a brand new and key new NFS so it's gonna take a while for all those patches to land and make it into distributions. So we really like this model for a couple of sort of key reasons. One is security, the tenant remains isolated from the storage network and there's no shared IP network between the host and the guests. You have to worry about this accidentally exposing host services to the guest network. It's very simple. So there's no network configuration aside from the host just assigning Vsoc addresses to each of the virtual machine. It goes back to a model where you treat the virtual machine as a black box. So there's no configuration that has to happen. It'll either use that Vsoc address or it won't. And there's no software defined networking, you don't have to worry about neutron or anything like that involved. We like the reliability model because the gateway that's sitting on the host is in the same hardware fault domain. So if that node fails, the guest fails too and you don't really worry about dealing with them failing independently from each other. There are fewer network traversals, so the IO stays on the host and then goes to the distributed file system. You don't have to hop to a service VM that might be running on another machine in your cloud. Which means that the performance is gonna be much better. So it's a clear win over having a service VM where you have multiple network hops. It might also end up being a performance win over using TCP, although currently we're not optimized for that. The Vsoc driver and Kimi users for IO but it's not really optimized for performance. It's really about configuration and simplicity and security. But there are some challenges. So Vsoc is a new hotness, it's brand new, we have to get all this code upstream. So it's sort of work in progress. There is some host configuration that has to happen so the host has to decide what the addresses are for the VMs. And someone actually needs to go and do all this configuration so they need to mount Ceph on the host and set up the NFS export to the guest to actually make this happen. And so the big question for us is what is the user experiment experience gonna look like? So does the consumer of the Manila API, are they expected to just know that this is how everything's gonna work and that they're gonna have to mount this magical Vsoc address in order to attach to their share? Or are there gonna be new APIs that do some of this work for them and then tell them what the mount mechanism is gonna be? So that's the big question. Which brings me to one of my final topics which is really how the responsibility for this is gonna break down between what Manila's doing on the sort of the file share side and what Nova is doing with all these virtual machines. So Manila obviously is managing the shares and volumes and Nova's managing the virtual machines. I think that the cleanest analogy to look at is the Cinder Nova breakdown. So in Cinder, it also just manages the volumes. Nova manages the VMs but Nova has this additional set of API calls that let you attach a Nova, a Cinder volume to a virtual machine because that's driver dependent. It knows how to do it with KVM but maybe for other hypervisors it doesn't. So we think that the same model should work well for Manila. So what we'd like to see ideally is a new API in Nova that allows you to attach a file system, a Manila file system to a particular guest. So in this case, the hypervisor is the one that's sort of mediating access to that file access share. It might do some network configuration like attaching to the Neutron share network that Manila's created. It might be assigning the Vsoc address and setting up this NFS re-export. It would be totally driver dependent in that case. It also is gonna be important to figure out how this is gonna work in a container environment where it's not KVM at all but it's no LXC or something like that. We also think we need to have a second API call that lets you fetch the access metadata which tells the user how they're gonna mount the system. So which Vsoc address they need to mount or whether they should mount over the network or whether it's a bind mount from something inside a container. And because Nova now understands which file systems are gonna be attached to the tenant, it can manage things like reattaching these file shares after you reboot the system or if the Nova VM gets migrated to another host, it can also do whatever configuration is necessary on the new host in order to make that work. So we've made a chart of what all the different Nova drivers would actually have to do in this case. I'll just look at a couple of these. So in sort of the traditional Manila model where we're exporting NFS over the network, the attach might just take the virtual machine that you already have and connect it to the neutron network that the Manila's managing. In the case of doing this Vsoc thing with Ganesha, the attach API would actually set up the Ganesha server on the host, assign a Vsoc address to the VM and do that NFS export configuration. And in the case of sort of a completely different environment we're using one of the container Nova drivers like LXC or LXD. It might be to do something completely different. Like it might mount the stuff file system on the host and then just do a bind mount in the container name space so that you can access that file share. And it's because the way that the guest access as the file system is completely dependent on what the Nova driver is that Nova really needs to be involved in this whole process and do that arbitration. Which brings me to containers which is sort of the last little bit here. So our vision here is that when you're using LXC or LXD in a Nova environment that you want something sort of the ideal model here to plumb that file access to the guest is something that's very different than what you use from KVM. You don't necessarily want a container to have a new NFS mount to the file server. There's a much more effective way to do that. So the idea is that you would mount Cep or whatever other file system on the host and then just do a mount dash bind somewhere into the container name space. So our thought is that you would actually mount this to something in slash dev. And then the user would do the final mount to whatever their final location is within their file system so that they can attach that file share where they want to see it. The pros here is that you get the best performance using sort of the native driver on the host. Containers are all about sort of cutting out the virtualization overhead so that's all good. And you also get the full set of S semantics with like snapshots and so forth that I'm through the command line. The only real drawbacks here are that you're relying on the container for security which is sort of obvious since you're using containers in the first place. But in order to make this model work you really need that Nova attach detach API because doing this plumbing is entirely dependent on the Nova driver that you're using. So in summary, the Seth native driver for Manila is something that we've created, it should land soon. We hope it works for us and the next version of Seth jewel which comes out quarter one of next year is gonna have a stable production ready Seth of S. So we hope to have sort of a complete solution where we can use Seth and Manila in real cloud soon. The current Manila drivers are sort of centered around appliances and are also assuming sort of a virtual machine network file system access model. Don't really contemplate containers. We think that NFS over VSOC is gonna be a better way to plumb file access to containers. We think it's very promising for simplicity reasons, reliability, security, and performance. And there's a lot of choice there. You can use the NFS kernel server, Ganesha, whatever stack and we find it to be most reliable or attractive. But in order to make this happen we really need to sort out this interaction between Nova and Manila and where the responsibility breaks down in terms of connecting this file shares to the guests. As we think that those Nova APIs are gonna help us handle the non-KVM users, particularly containers. They might even help in the ironic case. But most importantly, it'll allow sort of the right person, in this case, Nova be responsible for setting up that gateway that re-exports NFS to the guest. So that's it. Thank you very much. I'm happy to take any questions. Sorry, I'll just repeat the question. When you connect or you export something over VSOC does the guests get notification? And the answer there I'm pretty sure is no. In fact, it's analogous to a server exposing a service on the other end of the network. You don't know until you try to connect to it. Yes, so there's still the last step of the user tenant actually mounting the file system is still sort of left up to the user. And I think in general there's a lot of reluctance to have any of the open stack services reaching inside the container and doing any work. There's some agents that do some things that, but that always I think has to be an opt-in behavior. So I think the approach that makes the most sense is to make sure that there is a user-facing API that gives the user all the information it needs to know how to do them out. And then they can do that final step. With all the dual goodness and server face, are there any caveats to running the rest of your cluster on Hammer or something? So can you have dual server face with everything else on Hammer? You definitely need the dual stuff metadata server. I'm trying to remember if there are any dependencies on the OSDs. I think currently there aren't, but by the time we get to dual there will be. We changed the way that object enumeration works so that the recovery tools work properly. And in general it's gonna be something that you would do better off doing anyway, keeping everything on the same version. But dual is gonna be the next LTS release. So it's probably gonna be the version that'll be in 1604 Ubuntu. It'll be in the next Red Hat storage release that comes out next spring. So by that time you're probably gonna wanna upgrade to dual anyway. Back here. What was the one that you were supposed to talk about? Sorry? Sorry? OpenSack Magnum is the question. So I don't know a whole lot about Magnum, but my understanding is that Magnum is about wrapping Kubernetes. And then Kubernetes would be orchestrating containers. I'm not, my understanding is that those containers are actually running inside Nova VMs or on top of Ironic. It's not actually orchestrating containers that Nova itself is managing. No, that's not entirely correct. So with Magnum the containers are not managed by Nova. So you did talk about integrating with Nova. So the question is whether you guys have any plans about integrating with OpenSack Magnum, which does not involve Nova. On the storage side we have no plans. So I don't know what the larger strategy around container orchestration is. It's not really my area. I think if you are doing it using Nova to manage those containers, then all of the stuff is just gonna work. If you don't, if those containers exist inside of existing Nova instances, then again we would use whatever methodology makes the most sense to get the file access into the Nova tenant. And then you would have to do some additional work to get it inside the Magnum container. Yes? Do you have any plan to export HDFS through Manila and integrate with Sahara project? We don't have specific plans around that on the Cef side. My understanding is that there's already an HDFS Minilla driver. In fact, there's a talk I think yesterday about doing just that. So I think you can already do that. If you wanna do all that on top of Cef, run HDFS on top of Cef. I don't know, I don't think you wanna do that. So no, I guess is the answer. Yes. The question is about a CefFS Windows client. One does exist. It uses a framework called DoCAN, which is sort of like a user space file system framework for Windows. It's out of tree right now. It's a separate project, although it's on the Kithab Cef group repository. There's a lot of work that needs to be done to sort of bring that fork in line with the mainline code. So, I think it works, but I've never tried it. So, your mileage may vary. Yes, back here. Hi there, is this gonna cause any problems? Is Minilla gonna cause any problems for the RDMA project with Cef? I don't think it should make any difference. Well no, if you're using the Cef native driver where the tenant VM wants to mount CefFS directly, RDMA is not gonna work in that context, I assume. If you are gonna do something, like the VSOC model that I'm talking about that we're looking at, then it's just a matter of whether the host kernel supports RDMA, because it's the one that's actually mounting CefFS. So in that case, it would work. And I think that would be true whether using the NFS, or the kernel client in kernel NFS, or whether using Ganesha using libcffs. In both cases, it's really a matter of what the host operating system and kernel support. And the VSOC part is sort of separate. I think that folks grow, and now cloud-press is becoming stable. It would make sense that Nova will use CefFS to enable live migration. Yes, and in fact, that's one of the reasons why we think Nova should be the one that's managing these attachments, cause you want to be able to attach a file system and then live migrate it to another host and have it still keep working. And if plumbing the file access to the VM involves this sort of complicated going through a gateway that's on the host and Manila's trying to push configuration to the Nova hypervisor to do that, it's going to break for the ephemeral drive. Like the boot volume? I'm not sure I know what to send the question. Okay. And then you used QCAL 2. You could do that. That's probably not what I would recommend, although I know some people do do that. So yeah, any other questions? Yes? Which replication? Oh, geo-replication. Not initially, no. That's gonna be a ways off before that's supported. I think eventually we'll want to look at that, but we're focusing initially on doing geo-replication in the RATOS gateway for federating all these object gateways and on the block device for replicating block devices, which will be in the dual release. But for CefFS, we're not looking at geo-replication yet. Yes? I assume there's a monolithic, but honestly, I don't know. I'm not the open stack expert around here, it turns out. Yes? CefFS are for Windows guys. We're talking about file access for Windows clients. As I mentioned before, there is a CefFS client for Windows based on the DoCAN framework, but it's out of tree and we haven't tested it directly, so I'm not sure how well it works, but I believe it does work to some degree. Over VSOC. That I'm not sure about, but I'm guessing it's possible because this VSOC type is actually many years old. I think VMware added it to VMware, what ESX like five years ago or more, and I got it upstream in Linux maybe two years ago, and my guess is that the whole point was to do stuff like NFS over VSOC to the server, but I'm not certain. So it's a really question of whether the Windows and NFS clients support VSOC, and that I don't know. Okay, thank you. Thank you. Anything else? All right, I'm gonna post these slides on SlideShare and the last slide I have a whole bunch of links if anybody wants to go look at the code that I'm referencing in patches, so thank you very much.


Good day, everyone. My name is Pashem Kushufa, and my work is a call for actions to eradicate the precipituri mina, or PPR, as the solution to improve the livelihood of smallholder farmers in developing countries. And we draw evidence from Senegal. Meet Fanta.  One of the 300 million smallholder farmers worldwide, mainly women, who depend on small remaining for food and income. The animals are the Ateans. Every year, the PPR virus, which has 90% mortality rate, is killing millions of sheep and goats and hampering the productivity of those that manage to survive. The virus is a rotten smallholder farmer, like Fanta, of their basic right to good nutrition and livelihood. But it does not end there. In this study we conducted in Ethiopia and Burkina Faso. We saw that the impact of PPR goes beyond household like Fanta. The World Organization for Animal Health or E-estimate that the global damage due to PPR annually is up to 2 billion US dollar, turning back the clock on decade of developing progress and outcomes. The good news that PPR can be prevented with an effective and affordable vaccine that gives lifelong immunity to animals. Globally, the target to eradicate PPR is for 2030. In Senegal, that is 2025. Yet, PPR vaccine shortage is a critical issue in the Sahel and in Senegal. So to ensure that farmers like Fanta are part of the solution to this challenge, we run participatory disease modeling sections and we collected various data on herd size, vaccine availability, animal mobility, and we mapped it along key natural resources. This helped us to develop a special risk-based criteria to prioritize vaccinations. Later, we will assess the benefits cost ratio of these targeted vaccinations. Again, the standard practice of blind mass vaccinations campaign. What we know from our research, however, is that globally, the benefits cost ratio of PPR eradications is $33.8 per dollar. We can therefore expect more significant return using a targeted approach combined with biosecurity measures for instance. So investing in PPR vaccination is the viable solution to this global disaster. We are partners, we are then raising awareness and calling for actions to support PPR eradications in the Sahel and world world, to ensure food and nutrition security, poverty and conflict reductions, and to strengthen the resilience of these 100 million one-quarter farmers, just like Fanta and her children, to give them a better life while also achieving the sustainable development goals. With that, I thank you for your attentions.


My name is Franny and welcome everyone to Liquid Margins. This is social annotation online and on campus. Today's guests are Mary Isbell. She's associate professor of English and director of first year writing. And sorry, I neglected to put your university. I'm not on my game this morning. So it's international professor of English. I'm at the University of New Haven. New Haven University of New Haven, right? I shouldn't have my head start. I think my, this might be decapped by accident. I don't know what's going on. We also have John Stewart and he is the assistant director office of digital learning at University of Oklahoma. And then on moderator today is Jeremy Dean. He's VP of education at hypothesis. And we also have a couple other colleagues from hypothesis here, Becky George and Nate Angel and myself. And with that, I'm going to turn it over to Jeremy. And he's sure to do a better job. I've just done so. But anyway, happy Friday. Thanks for being here. You're doing great as always for any. I'm so excited to be here. I know both Mary and John from before the pandemic. This, this session is called social annotation inside and outside or off campus and on campus, but the secret title is, you know, like social annotation, it's not just for pandemics and we know each other from before the pandemic. I've met you both in person. We've both presented together.  Have you met before Mary and John? No, I don't think so. Okay. Well, you're two of the oldest school advocates of hypothesis at different different corners of the world. You know, Mary's been using hypothesis probably since before we had a learning management system integration, I think, across two different LMSs now at the University of New Haven from Blackboard to canvas. And John, certainly before the LMS as well, I remember we had, you guys integrated hypothesis with wordpress there. You're enough of a developer that I know you helped us start to explore certain ways of viewing data even before we, we had kind of dashboards for users and stuff like that. So you both have been around a long time with hypothesis, which is why we've invited you here today. And I'm very happy to see you. How have you been? Where you want to share a little how you've been? Yeah, I'm surviving. I'm joining you from my mother's apartment attached to my home where my two children are over here. Hopefully we won't hear them screaming. I feel like that's an important part of sharing how I'm doing. I don't have a care still. The semester is almost over and I'm very excited about spring and summer. So yeah, I'm doing great though. John, how are you doing about the same for me. My daughter's right behind me eating lunch already. And I've been home with her for, you know, a year now and just feed her constantly and do dishes. And then occasionally get to work on school stuff. So John, you've been working from home this whole time. Yeah, I had been all calendar year 2020 this year. I've actually been back in the classroom and I've been using hypothesis face to face this semester. So that's been interesting. You know, in addition to all the times in the past. And Mary, are you. Have you been going to campus to teach? No, so our situation made remote teaching the smartest choice since we were trying not to expose my mom to the virus. So I taught remotely for this entire year. And actually the spring semester, I wasn't teaching because of maternity leave. So I hit the ground running in fall, but it wasn't part of the crisis of the last spring. But yeah, it's been, it's been all remote and hypothesis has helped a lot with that actually. So congratulations.  Yes, I have. He's almost 16 months old. Congratulations. I wanted to start off by reading this quote that I actually got turned to by Joshua Kim and his regular column in inside higher ed. It's a quote from a Peter Bryant, who's a professor and associate dean at the university of Sydney business school and a blog that he talks about what he calls the snapback, you know, personally, familially, all different, you know, athletically, maybe academically, professionally, there's a lot of desire to return to normalcy, right? I actually just got my second shot this morning. I'm feeling fine so far. And so I'm excited, you know, the person who gave me this shot asked like, do you have any travel plans this summer? And I was like, well, I don't know. I sure would like to go to Maine and visit my family and go to Minnesota, visit my wife's family, but I don't know. But we do have this desire to return to normalcy and he calls that the snapback. And what he's trying to do in the blog is kind of ask us to pause and reflect on what we've learned during this time that might stay with us beyond the pandemic, you know, like maybe the N95s we can at some point burn them. I think probably not. But, you know, certain things we will stop doing, but certain things, you know, we will stop doing that. And I think that's what we're trying to do. And I think that makes sense. Actually, I was standing there in line today for the shot. And I was looking at the stickers, you know, the six feet distance stickers. And I was just thinking like. Everybody who's been alive at this time is going to know like. Six feet really clear in their head. You know, they've just been standing at six feet distance for so long. It's going to be a measurement that we just kind of automatically recognize. I want to read this quote. And then we'll now turn into this more of a conversation. It's about knowing what we have learned in our liminal year, year or two. This isn't about case studies, conference papers or vendor demonstrations. It's about knowing the human impact of what we have all been doing. It is understanding the affordances of a horrible situation and knowing what we have learned from experience and telling those stories to the right people. At the right time. Not when we can sort of participate in that. Today. I saw a post this morning of a Twitter. And it was published by Paul Manson, who's a visiting professor at read. And he says tools I've used during the pandemic that I want to keep. And it's got some checklist. I don't know how you do this in Twitter, but it's an in class chat software. Allow them all to speak in ways that work for them. Zoom office hours, lower barrier to check in. And then a thirdly online reading discussion tool, hypothesis for sure. What tools have you and I would say tools and practices. What tools and practices have you discovered or rediscovered over the past year. That you feel like will be part of your repertoire when you, we hope, return to normalcy. Mary, you want to start us off? Sure. So I think I, we did transition to canvas in the summer before this academic year. So some, a lot of things I've changed had to do with transitioning into both using canvas and teaching remote synchronous. Hypothesis was one thing that I was now using inside the LMS. And so I think I, I upped my game in certain places there using canvas. I developed a couple of new strategies to have students return back to the annotations that they've made. Another tool that I'll throw in the mix though would be any sort of collaborative writing environment. I switched after taking an excellent course on data code and ethics during the digital, digital pedagogy lab. I made this switch away from Google docs. I'll share that here and transitioned into using our own universities, Microsoft OneDrive system to use shared word online documents. Either way, I had always used them to do one off revision workshop activities in class. But what happened this year is I just created one document that we used all year and the new content went up at the top. And the way I was using that was in tandem with Hypothesis. I was having students pull things that they had cited and talked about in that social, social annotation space, pull them into the shared document and then use that as the space where we communicated with each other. And it felt so frustrating at first to not see their faces until I adjusted my expectations about why cameras weren't on and all the reasons that one might feel exhausted by having Zoom on even if one were in the perfect circumstances. That I started to just say hello at the beginning and then everybody's cameras were off and we were in this shared document and then cameras would come back on at the end. And that has proven so productive. We get so much done, so much synchronous discussion happening that I've really enjoyed that and I'm going to keep it. I'll stop there for now. I'm sure more will come up. Yeah, I really like that idea of the shared notepad, the shared, you know, class notes. I haven't tried before with just a single document for a whole class. But yeah, I was doing similar stuff with my class and in Google Docs, having shared notes for each week usually. And for me, that's part of how Hypothesis sort of fits into what I'm trying to do is shared note taking and thinking out loud more broadly. And so the other piece of that for me is blogs. And so I've got my students set up a blog with each of them and they blog weekly. Either what I did was I gave them sort of an assignment bank and I said choose from any of these prompts reflecting on what we were talking about this week, bringing in outside sources or outside articles that they read, talking about their favorite music or movie or anything else that they're engaging with or, you know, about a half dozen other prompts each week and just trying to get them to think about how what we're talking about in class relates to the rest of their world and reflects their own interests in the broader world. So that's been the main space for me. But yeah, I see, I see all of these tools is doing the same work of thinking out loud and trying to connect in class with authentic learning. I was going to ask what is the class you're teaching that you were describing that work in? Oh, sure. Sorry. Yeah. And we introduced me as assistant director for an office of digital learning, which is my job. But right now I also teach history. And so right now I'm teaching a history of science class where we're using hypothesis. And so it's an intro to history of science. And we've actually been talking a lot about public health this year and then the other class that I'm teaching, we're not using hypothesis in this other one, but a lot of similar work is on the history of the Tulsa race massacre, which is coming up on its 100 year anniversary on May 1st of this year. And so anyway, I can talk more about that. If anybody's interested in the history, if you don't know the story, but both those classes have been really interesting. Yeah. And one thing you're you're reminding me of is that other piece of that writing for the public writing for a real audience. Did you, which blogging platform where you having your students use, could they choose or what were they doing? Yeah, so we, we have a domain of one zone initiative here at OU where everybody can have their own domain space and build whatever kind of websites they want to. For my class, I was encouraging them strongly to use WordPress. And so each student has their own full WordPress site setup. And then we use canvases or LMS. And so they can submit their blog posts into canvas. And it brings it into the grader for me. And I can use the grader just to see what they've done. Yeah, just as an assignment type, you can ask for a URL submission. And so it brings it all in. So that's been really nice. I have installed. Well, I've got an installation going at the University of New Haven of open lab, which is commons in a box, but customized by city tech. And it's, you know, you may all be familiar with the sort of that. And it's because I so want to be a domain of one's own campus. And we can't, this is, this does allow students to create their own projects and associated WordPress sites, but it also means that I get to create a WordPress site for my class and students can join as members. And so a lot of, I was experimenting for the first time really with that as well, having students turn projects into public facing work, which was great. But I think part of what I've wanted to do with all of my classes for years now is just no disposable sort of assignments.  And so, you know, having them think about who their audience is having them write for each other and having them comment on each other's, but we don't actually use comments in the blogs anymore, but we use hypothesis and we use some other stuff to, you know, give feedback. But yeah, that idea of, you know, talk to talk to your parents, talk to your roommates, talk to your friends, talk to your friends. And just the broader world rather than to me. Yeah. Yeah, exactly. I think we're on the same page on that front. And I actually have thoughts about that. When you have hypothesis. What do you do with the comments tool? I don't find students gravitating to use it. On their blog posts, even potentially not making them, like not turning it on. You can turn it off on each blog post. So, yeah, really interesting. to be more deeply integrated with both those versions of WordPress so that your Canvas hypothesis login will work across those platforms. I'm wondering if each of you can reach back into your memories about what first attracted you to social annotation or hypothesis and talk a little bit about that and then also how your practice maybe has changed or evolved over the past year. You hinted at this Mary in terms of returning to annotations, but John, maybe we can start with you this time about what first interested in social annotation and if and how your practice with annotation in the classroom has changed over the past year during the pandemic. Yeah, it's a similar story sort of returning to what I'd been doing a little bit in the past. One of the things about history of science is that it's a smaller field of history. We don't have textbooks in the same way that American history course does or European history course does. And so while there are books that are commonly used, a lot of our resources that we give to students are articles or other online resources. And so one of the challenges of hypothesis is that it doesn't work as well with a physical textbook, but there's just not a thing in my field. And so all of the assignments that I gave to my students this semester were either PDFs or web posts, blogs, articles on NPR or whatever. And so everything was annotatable as far as our readings go. And so it just was a normal fit for me. It was a really logical use case for me. And that was one of the things that first attracted me to hypothesis is it just works really well in my space. And this semester, it had that same function. The other thing I like about it and the other way that I've used hypothesis in the past is sort of as a curation tool. So it's great for annotation obviously, and I think that's the main way people think of it. But as I'm going through and just reading things and coming across whatever in my daily practice, I can leave a comment and sort of collect that thing for myself. And remember, I came across one article on NPR where they were talking about genetic splicing of humans and apes of various sorts. And they're looking to be able to grow human organs basically on monkeys. We had just been talking about the Island of Dr. Moreau in my class, which has the same themes. And so I read that immediately thought of the priorities discussion and marked it and then passed it on to my students. And I had that sort of experience on a nearly weekly basis throughout the semester of just being able to pull things into my class and encouraging the students to do the same. I love that. Not a disposable tool for you either, right? As a scholar, as a teacher, something you're using and then connecting it with your teaching as well. Mary, what about you? What first attracted you to hypothesis or social annotation? And how's your practice evolved, especially over the past year in a new teaching context? Yeah, so what attracted me to hypothesis, I wanted something like hypothesis to exist before it did. I feel like I was, I had as a graduate student encountered in teaching first year writing courses. I was also serving in sort of a graduate student leadership role in the first year writing office. And I was frustrated by the anthologies that we often used when teaching our composition courses. We had two courses then, and I think they still do at UConn. One is you learn writing through nonfiction and one you learn writing through literature, quote unquote. And when we would teach that other course, you would often choose an intro to literature anthology or some kind of textbook that gathered together readings for students. And I was always frustrated by the apparatus of those anthologies. The apparatus got in the way of what I wanted students to do with them. They presented literature as content to be mastered, certain kinds of literary techniques needed to be learned and regurgitated. And that just got in the way. So I worked on creating an anthology with my colleagues, my fellow graduate students. And that anthology, I won't go into the details, but I worked on trying to publish something that would pull it all together nicely but have the apparatus we wanted in. It wound up despite our best efforts costing a ton of money. And it was so frustrating because I knew that most of the readings we wanted to choose, some were under copyright and that was a different story. But most of the readings we wanted to choose were in the public domain and it shouldn't have cost anything. And there should have been a way for us to be able to give students a way to engage. So I first was using Annotation Studio that I learned about that MIT had developed. And that's actually what led me to meeting Jeremy and learning about hypothesis. And so for me it became, this will be my non-anthology way of gathering readings. And it worked really well. I always wanted students to read and respond. My goal in the writing classroom at least is to have students engage on the level that they want to engage. So what surprises you, shocks you, any reaction you have, my goal is to teach them how to take that and use it to develop an original arguable claim and to recognize that the thing they think can develop into something exciting to say, not just wait to figure out what everyone says in class to figure out what the right answer is. So I've been using it that way. And the way things are changing now that I'm really excited about actually is I'm starting to realize that that can be the first step toward collaboratively creating an addition with my students for future students. So this, all things are kind of meshing together. But this idea of I don't want disposable assignments. I want the writing that my students produce to wind up somewhere is coming together with annotation so that I'm kind of thinking now, not kind of I am pursuing a project where students will write initial annotations in hypothesis, revise them in hypothesis and then submit a formal assignment which is a sort of focused annotations in a particular area that they think will improve our classroom addition. And then during the summer or the winter break I'll incorporate that into an addition that moves forward and it won't, that then won't be in hypothesis. So it'll be in the actual text. And so that's a big thing that hypothesis has facilitated. I wouldn't have thought about things in that way without it and it's gonna be an important tool to develop it for sure. Are you thinking about centralizing those texts that the students collect into sort of a digital anthology or are you thinking about leaving them sort of spread out all over the web but then using hypothesis to like string it together in a web? I like that question. The project started thinking I'll be making an anthology. And I was thinking about it in terms of some of you might have come across this initiative at Purdue called Cornerstone, Living for Learning. It's an initiative to, it's NEH and Tegel funded and it captured humanities professor's attention across the country for sure because it was here's funding to develop an integrated core curriculum experience in the humanities for schools that have pre-professional programs primarily. And that's our instance at the University of New Haven. I teach core courses to students who are not English majors. So I got really excited about that program and part of their program at Purdue at least has a set of transformative texts. And so my initial idea was to develop an anthology of transformative texts that we determined together we would work with. And as I thought more and more about it, I realized my pedagogical, my approach to texts says it's not so much about what the text tells you it's about what you do to the text. And so an anthology says these are the things we find valuable for whatever reason even if they're troubling or what have you. So long answer to your very simple question. I'm thinking more that it's distributed and I'm actually thinking that I'm one person creating additions with my students but the project's morphing into being more here are the resources that any instructor can use to create additions with their students. And there will be a hub to sort of share additions if people want to make them available to others to use. But mostly my dream would be that people would get the bug and start thinking of editing as a collaborative effort to undertake with their students. I wonder if you guys have noticed any difference in how your students have responded to hypothesis during the pandemic. When you've taught with hypothesis in the past they probably still met face to face and got to socialize and connect and converse and discuss course content then. But when we remove some of the more, you know the social aspects of teaching and learning if you are teaching face to face did you find students more open to the tool or more dependent on it in any way?  And I think even without the social distancing they see each other's comments in the readings and respond and start having conversations there with someone that they might not have talked to prior in the semester or someone they might not have connected with. And so I think I do like that hypothesis. I like that there's no avatar I guess in hypothesis that you don't necessarily know what the other person looks like or who they are necessarily in the class.  And so as a social media tool in that way I like that the conversations there are sort of have a different context I guess than what they're having in the classroom. Yeah, grounded. Yeah, that's great. Mary. What I used to do in the in-person classroom was have them all annotate and then I would kind of gather some notes of what I was noticing. And then the beginning of class was, okay, these are some threads, let's go. And then it would almost be like that avatar would meet the person and they would say, oh, that was you saying that and that was you saying that and that would sort of make it come alive. And that didn't happen in the same way but because I would have them move from hypothesis into pulling the thing they, so they might annotate 15, 20 passages in a reading that we were doing but I'd ask them to go back through what they had looked at and pick two that they really wanted to sort of share what they thought about it. And that then goes in the sort of shared word online document space. And they're their name show up, what they're putting is under their names and they start to comment on each other and that's the live activity during class. And yeah, it's not the same as in-person but it's still, I think had them start to develop this or recognize the personalities of what was being pulled out, I guess. Mary, can you say more about this assignment in terms of, so I've annotated 15 times in a text and then the next step is to grab my reference and my annotation and bring that into the Microsoft word space. So it would be some of the original text, my comment. And then I say more, I like elaborate. Yeah, elaborate. Basically it wasn't even so much, I gave them the option to sort of say again what they said in their annotation but I would frame the discussion in a certain way after having looked at how, so for an example, I was using this, not in a composition course, this activity but for a course on review writing, the art of the review, reviewing films, reviewing television shows, et cetera. So we would read reviews that we decided on together. We'd found a review of John Mulaney's comedy, Netflix special, so they'd all read it and annotated it with hypothesis. And then in class, I wanted them to isolate some moves that the critic was making in writing about Mulaney. So they would under their name share a quote and then identify what they thought the rhetorical move was that the author was making. And then the next step of the activity was for them to respond to each other. And that's where the discussion got cool, right? Because the comments in the document was where the back and forth was happening and everyone's together. So there's hypothesis annotations as homework where they may be in the document at the same time or not. And there's also hypothesis annotations live during class which I had also done. And this kind of brought pieces of that together but because they were all in there together all the typings happening at once. It made up for the fact that I didn't see faces on screens and there was action. That's super cool. One strand that I'm hearing here is that I really like is just the idea that an annotation is in an end in itself that it's the beginning of something that continues in face-to-face class time when some more collaborative authoring tool like you're describing, Mary, that's really cool. I wanna open it up to our guests and also to our audience and also to my colleagues if you have any questions about what Mary and John have said. The crowd has been a little quiet today in the chat. People have been commenting about how much they wanna take classes with these folks because the classes sound so interesting. We don't have any active questions but I know you probably have some more of your own Jeremy. I have some as well, but I'll let you go first. I mean, let me just get the sound bite in there. Like social annotation, is it just for pandemics? I mean, hypothesis over the last year has seen tremendous growth, tremendous adoption. A lot of new schools have gotten you guys been around forever and University of New Haven is actually one of our first customers. Oklahoma is now a piloting for the first time but there's a decision coming up at Oklahoma, right? Should you guys subscribe? I don't have to put it in the context of your specific school but there's a lot of decision makers and teachers themselves that are sort of deciding with this snapback, with the return to normal. Now that can go into face-to-face and see my students for those that teach face-to-face, do I need tools like this? I'm assuming the answer is yes, but I guess the question, the follow-up is walk. And if you want to say no, you can. Yeah, I mean, for me, it's always been about, before COVID and still having the discussion at the level of the reading and not waiting until sort of the end analysis. So yeah, from the researcher side, it was always about thinking out loud and discussing stuff as I was reading it before I get to the stage of writing an article or trying to put it in the context of like a broader book project or whatever. And then with my students, being able to point them to specific passages and then see their reactions to specific passages. And so it's a tool that, yeah, I will continue using after COVID and after everything else. And then I just, I keep thinking of other use cases in other classes. I'm working with one class, it's actually an architecture and they're doing a oral history project where they're taking all of these oral histories of Oklahoma City and they're coding for locations and affect of those locations within the oral histories. And hypothesis makes that really easy. You just highlight a thing and then you can enter in the geolocation or description of the location that's mentioned. And so it's, whether you're doing that asynchronously, synchronously, face-to-face or online, the work still needs to be done and hypothesis makes it easy. So yeah, I hope, I know that I'll continue using it. I hope that the school continues using it. The argument I think for the LMS integration, the cost to an institution being worth it has to do with the fact that I'm this sort of weird professor who was seeking this tool for a long time and like an eager early adopter and happy to experiment and happy to deal with the discomfort of students when they're trying to get a new thing set up when it was a hypothesis in the wild, right? And I could put up with that. There are so many of my colleagues who are more hesitant about using technology and who at the same time would complain about students not reading would say, I give them these assignments and I don't think they're reading them. And I think I could say to them, I don't think they're reading them either. You don't have a good way yet to hold them accountable for reading, not only hold them accountable for reading but give them a way to read in an engaging way. Like, it's not about like replacing the quiz with the annotations, it's about giving them this experience of the value of reading and that's something that's gonna be done with what they've come up with during class. It's not just that the professor's then going to lecture all the things that were in the reading. So the reason I'm sharing all of that is because the value I think to an institution is that the instructors who might be a little more hesitant to try won't have as many hurdles when using the integration into the LMS. And I've talked to many who've given it a try. They put up a PDF students annotated and they're blown away by the level of discussion that was occurring. And that's the beginning of a flipping of the classroom. Even if that instructor hadn't already drank the Kool-Aid of a student-centered classroom where there is no lecture, you're gonna get drawn into that because you're gonna see everything your students wanna share. So from the perspective of just professional development of faculty, I would say, I think it's valuable there. And I don't think you'll have as much experimentation if you don't have the app in the LMS. There's a little humorous. Those are great points I thought. There's a little humorous chatter going on in the chat now about what's hanging on both of your walls. John has apparently the world's largest diploma. Well, maybe not. There could be people who have even larger diplomas because they have larger walls based on how he says it works. Mary, what is your thing on your wall say? I have been so self-conscious of my location for teaching because this is my mother's apartment. She'll be happy to know that this will be on the internet one day. So I have told every class, especially in the fall, I was so hyper aware of this being in the frame that I would say to my students, can you guess what it is?  Are there any guesses? Nobody ever took me up on it. It never became like the buzzy thing in my class that I wanted it to be. But does anybody have a guess? This is giving me a... There's a little bit of glare on it right now. So it's a little hard to see. Hey, I mean, this is basically like, I gave you the perfect amount of glare to make this really hard nut to crack here. If I were to tell you that this said what a... Word? Word? Wonderful world. A? Yeah, so... Is that from the song? Yeah, you could imagine that the decor choices of my mother might involve Louis Armstrong. Oh, I'm a big fan, love that. And so it guides our day. Nice. So that's what we've got going on. Thank you for asking that because I wanted somebody to talk to me about that since I started teaching here. That was a very important question, Nate. Yeah. I have another follow-up question that's a little more serious too if you will allow, Jeremy. Thumbs up. You guys may have heard, I'm gonna throw a link into the chat here about the research project that Hypothesis has launched with the University, or Indiana University, sort of thanks to a lot of guidance and work by our scholar-in-residence this year, Rami Kalyar, who's at the University of Colorado of Denver. And this will be certainly near and dear to Mary's heart and John's too, I'm sure, because the research project is designed to explore exactly the kind of intersection between kind of first-year reading and writing experiences like in the composition and English field. And I'm kind of curious, if you haven't heard about it yet, this blog post kind of lays it out, but just given the context of your work and I'll go to Mary first on this, would you say that you find, and I know you've addressed this a little bit already, but would you say that you find with your students that social annotation ends up being something that empowers writing as much as reading? Is it both or is it mostly about just the reading? Well, the reading is the beginning of writing. And so in that regard, I mean, the way I teach writing is to try to empower students to trust their impressions because in order for them to learn to develop original contributions to make to scholarly conversations. And that's what I'm driving at when I teach academic writing. Then I need them to trust that their impressions upon reading something, whether it's a scholarly article or a work of literature has value and not defer to my sense or whatever they can glean implicitly from class discussion about what am I supposed to know about this? And so I find student writing extremely stilted when they come in often from having been taught in whatever ways they have that there are wrong or right answers and that you have to convey certain knowledge of certain techniques. I have to undo all of that and social annotation helps me do that. Classroom discussion helps me do that but social annotation shows them as it's occurring, the variety. I used to be, I'm gonna go into the weeds on this. Stop me if you need to redirect but I used to be a little concerned about students being able to see each other's responses before class discussion because I used to think that the power of the class was the moment when the student said, I kind of thought this and then the student across the room said, I don't respond that way at all. But what I find in that moment, you can see the multiplicity of interpretations that are out there and I can model in the front of the room that could become an interesting claim. That could become an interesting claim. All of these things can be alive and that's exciting. I used to worry that having them all annotate together before class, they'd all just pile on to certain ideas and I've experimented with that. I've had them keep their annotations private at first and then share them in some way. But the more I teach them and the more I talk to them about what it is they're doing that they want to try to contribute something and not just jump on and say, I agree or push them to say, I agree but with a difference or I disagree in this way. All of that drives toward them developing claims that are arguable. It's absolutely crucial to the way that I teach writing. Yeah, I think that says it better than I could have but I did want to point out that the two biggest user groups at OU for hypothesis are creative writing and the history department. And so it's the two sides of that coin of annotating the historical texts that the history department is reading largely and then the creative writing using it to annotate. And all of our creative writing courses are taught using various sorts of literature and so they have the students engage with that literature and then write off of it. And so they're marking up both what they're reading and what they're writing. And then in my own classes, history of science is largely populated for the students by pre-meds and by other sort of STEM majors. And so they take our courses and we're expected to help them learn how to communicate with each other and how to write from a humanities perspective. And so hypothesis is great in helping me get them to analyze texts but also start that process of writing collaboratively and individually. And so yeah, I think it's a great transition tool in terms of seeing the connections between what you're reading and what you're writing. And I actually want to share something building on this because there's an activity that I built this semester that I think works exactly to, I'm so excited about this research project. I didn't know about it but I'm excited to learn about it. One of the things, as I was just describing, having students respond initially and then helping them recognize that what they first were thinking as they were reading can be the beginning of essentially a question that then they answer and that becomes their argument for their paper. I've realized that they respond in so many different kind of categorizable ways. Some respond with anecdote. They say, this reminds me of this thing that happened to me one time. Some respond immediately with a claim. Some respond immediately to a text by saying, I would argue this about it. Some respond with questions. Some respond by having had a question and already answering it by looking something up, giving us a definition or a gloss on a term. And so when they do their first round of annotations and we come to class, I introduce the categories that I've come up with and I ask them to go back and categorize what they've done and tag in that way. So we've been playing around with tagging and helping that essentially what that does is it helps me teach them the various components of building an argument that you need all of those things and that some of them come quickly to some people and some of them take longer but they can watch what everybody else has done and see how they would categorize it. And then when I ask them to do another activity and I say, all I want here are objective observations like these are just that no one would disagree that this occurs in the text. That's something that they can recognize what that is because they've tagged what they did before to see where other students were doing that. Is that making sense? Yeah, that's super cool. That's great. I'm gonna jump in here because we're a little bit over time but if people wanna keep this discussion going, that would be great. I just want to be cognizant of people's actual time and if you have to jet anywhere. So it's up to you. And again, also to people who came here today to see this, if you have to go, there will actually be a recording of this in the coming days, hopefully, so we'll reach out and share that with you. But John and Mary, do you wanna stay over a little bit? Keep the discussion going? I wanna share our last thought. You just, I think you just went the wrong way with me. I thought you nodding. I unmuted myself to tell John that I thought he should go first because he seemed like he had something to say. Oh, I was, so in addition to teaching history courses, the main thing I do is help everybody with their blogs here and here at OU. And so I work on a lot of web and DH projects and I was just imagining different ways of building this anthology, just using the hypothesis API and how you could have a sort of live updated table of contents with all of the stuff that your students are annotating, just pulling straight off the hypothesis API. And I was imagining to build on that. I would love to hear what you're thinking about that because you're talking about the project I was describing. Yeah, and just the way that you're using tags and the way that the students are going out and collecting things and, you know, whether that's directed by you or sort of in the wild, as you said, by the students, you could just live pull off of the groups tags and then, yeah, whether that's literally a tag cloud or sort of an automated table of contents of all of the different pieces that they're pulling together and then just linking out to wherever that is with the students annotations, you know, sort of live on the web, but your whole like anthology could be a single webpage that's just live updated, pointing out to all of these different resources that your students are collecting and annotating. I like that a lot. Would you like to join the project? Please, Sean. I think I could still show you how to do that. It's been a little while since I played with the API. I was thinking that the John Udall could probably do this in his sleep, but when I think about it, yeah. When Mary started this project, she's like, who's doing crazy, weird, like funky stuff with a hypothesis and I sent her an initial list. And I don't think John was on it, but I was a mistake. John should have been on there. He's always been trying to break the tool in all the best ways. Yeah. Yeah, I want to hear these ideas because actually a lot of the stuff I was thinking would be kind of using hypothesis and then building something that could be annotated with hypothesis later, but not necessarily thinking about the tagging in that way. So that's really helpful. And I'll be bothering you, John. You can expect it. Yeah. I think Jeremy has probably beaten this horse beyond death, but the metaphor of the mimics and the idea of showing our work as we read through lots of different sources and those connected paths. And it sounds like your anthologies is very much in that vein. And so yeah, just being able to chart those paths around the web as they develop. And obviously the downside of that, of leaving things out on the web rather than collecting them into a single space would be if any of those resources stop being maintained or go down or whatever. But that's something that, you know, I heard all you could jump in it. I have a closing observation, which is that the blog post that I started off by quoting sort of suggested that the world, that people, that teachers are fatigued and just ready to return to normalcy and that there's going to snap back to something that was before. And that's not what's happening here. I see two educators here who have always been innovative, innovative, continue to be innovative in some ways have been energized and more imaginative or, you know, that at least has continued through this difficult time. So that's a very optimistic thing for me to see that great teaching continues, great teaching continues to evolve. And we've had a great conversation with two great teachers today and I thank you for your time. Thank you, Jeremy. Yeah, I want to thank you too. This has been great. And maybe we have to think about, there is no normal anymore and go forward with that notion and all. And maybe there never was. Maybe there never was, yeah, exactly. So, okay, great. Well, again, I want to thank you both for being here. Mary and John, this was a really great, the good margins and thank you for everyone who showed up and everyone at Hypothesis. And again, there'll be a recording of this soon and we'll share that with you and join us again on liquid margins. And again, thank you for being here. Take care.