progress in lowering the cost of doing business in Vermont. I'll now invite Commissioner Pichek to elaborate on how we were able to see such significant decreases. Thank you, Governor. Thank you. Well, thank you, Governor, and good afternoon, everybody. I'm excited to be here with you today to share some great news that relates to Vermont employees and employers and explain some of the great work that our department staff has done over the last five months or so. As you may know, the worker compensation system in Vermont traces its origins back to 1915 and operates under three fundamental principles. That employees are injured on the job, should be compensated reasonable medical expenses, and also for wage replacement. That employer compensation system is the exclusive remedy for employees that are injured on the workplace, and that all employers must cover their employees with worker compensation insurance. Now, this last prong is particularly difficult for smaller businesses and new businesses and businesses looking to expand. The cost of worker compensation insurance can be prohibiting, particularly in some iconic high-risk industries in Vermont, like dairy farming and log hauling. This is why the rate reduction that went into effect this week is such welcome news for businesses across the state. This year, the department took a fresh look at the rate filing. We hired a new actuarial firm that brought new perspective, brought new ideas that our department vetted and in most parts accepted. As the governor said, this resulted in a 3.7% reduction in voluntary loss costs, additionally a 7.6% reduction of voluntary market, but that 7.6% reduction of voluntary market was supplemented by an additional 6.7% reduction due to an elimination of a surcharge that are actually recommended and our department approved. So in total, we see a 14.6% reduction in that voluntary loss in that assigned risk market. And the reason that assigned risk market is so important is because that's where individuals that can't get insurance through the voluntary market, they may be new businesses, started businesses, they may be small businesses looking to expand, or they may be these businesses operating in high-risk industries. So these are people that are gonna get serious relief from worker compensation insurance costs. One of the things that we wanna highlight beyond just the $10 million that we'll save this year is that when you combine with the rate reduction from last year, Vermont employers will be spending $30 million less in worker compensation premiums than they did in 2016. Mentioning these high-risk industries for a minute, our work focusing on logging and some other high-risk industries this summer, along with Sam Lincoln, who did an excellent job presenting these issues to us and advocating for the logging community. The governor mentioned a dramatic decrease in the log hauler class code. This was from the great work of our department and our actuary. They discovered a way that we could combine the log hauling class code with a larger trucker class code that would mean that Vermont was able to rely on our own experience, that we didn't have to rely on a melting of experience from national carriers, a national industry. And because our experience was so good here in Vermont, that's what resulted in this dramatic 24% reduction in log haulers. So I wanna thank the great work of Deputy Commissioner Tris Rulo, Kevin Gaffney, and Phil Keller, as well as Jill Rickards here from our department, who worked very hard on the Worker Compensation Study and also worked very hard on this rate filing. A special mention, and Sam will mention this, but the Worker Compensation Work Group is something that was also established in the last few months. That's looking at ways that we can improve workplace safety in these high-risk class codes, particularly in the logging class code, non-mechanized logging, the sawmill industry as well. We'll take what we've learned from other states in Northern England, like Maine, where they have seen dramatic decreases in the Worker Compensation Premiums due to much safer work environments. So with that, I also wanna thank NCCI that worked with us during the process. I wanna thank the VIAA and Mary Eversol and her agents and her members that do a great job of educating businesses as to their responsibilities for Worker Compensation Insurance and ensuring that workplace has become safer. And at the end of the day, that workplace safety is the key to reducing the rates in Vermont. The Department of Labor has done excellent efforts in this regard, both with their enforcement efforts and then also with their project work-safe efforts. So with that, I wanna invite Commissioner Curley to come up and explain a little bit more about that. Thank you, Commissioner Pichek. As both Commissioner Pichek and the governor explained, safe and healthy workplaces contribute greatly to the reduction in workers' compensation insurance costs. Current data shows, on average, workers' compensation insurance companies pay out nearly $60,000 per claim in wage replacement and medical expenses over the life of a claim. In the past few years, in Vermont, we have seen a decrease in the number of first reports of injury, as well as the number of lost-time claims filed with the Vermont Department of Labor. The decline reflects a growing commitment on the part of Vermont employers to put in place effective safety and health management systems in the workplace, emphasizing a cooperative management and workforce commitment to preventing injuries. Enforcement and consultation also play an important contributor role in helping to achieve the reduction in serious injuries. In addition to the commitment of employers, which is an important tool in the declining trend of worker injury, the state's Occupational Safety and Health Administration Program, or VOSHA, as most of you know it, has proven another tool that has greatly reduced the number of work-related injuries and fatalities occurring in Vermont. By providing compliance, assistance, education, and through our voluntary Project WorkSafe program, we've been able to work with employers at their request to provide assistance in identifying and correcting workplace hazards. Project Workplace provides safety audits, program development, chemical exposure, assessments, and noise monitoring at the voluntary request of the employer and services are free and confidential. Reducing costs, keeping a safe and healthy workplace can help increase productivity, improve and increase employee morale, and enable a company to win over and retain their employees. Additionally, as we look to grow our labor force in Vermont, investing in safe and healthy workplaces is an important strategy that will allow us to attract more families to our communities and the jobs that we have available here. At this point, I'd like to turn it over to Deputy Commissioner Lincoln. Thank you, Commissioner Curley. Workers' compensation, insurance, and Vermont's forest economy can be prohibitively expensive, with rates ranging from $19 to $51 per $100 payroll. From 2012 to 2016, those rates rose 60%. And thankfully, they plateaued at this time and are starting to decrease. This can cause those who work in Vermont's most dangerous and one of its most important occupations to work alone in our most remote places. Employers who have the insurance find the cost to be unsustainable, particularly when there's been a significant market collapse for low-grade forest products. Vermont is a small state with a small number of small businesses, and there are times that this can be a disadvantage when it comes to insurance. The change-to-log hauler class code addresses a complicated rate-making formula that has kept rates high for these employers, even when this occupation has had a notably positive safety record by implementing a common-sense solution, merging log hauling of contract trucking. We have this significant 24% reduction in rates. Our department has actively engaged in efforts to address the root causes of high rates for other occupations in the forest economy, including plans for enhanced safety training, verifications that the safe practices have been implemented at job sites, and ensuring that the contractors hold the proper insurance. Our overarching goal is to increase the number of employees under the protection of workers' compensation insurance, making sure they are better trained and working at safer job sites. When combined, these efforts will sustainably lower rates and reduce one of the well-known barriers to employment growth in this sector. The department has received support in this effort from Governor Scott, the other departments mentioned within the administration, the legislature, logger training programs, not just in Vermont, but from organizations in Maine, New York, New Hampshire, and all who have lent their expertise to us, and we are grateful to them all. I'll turn it back over to Governor Scott. Are there any questions about the issue of workers' compensation rates? I was going to have data showing that work sites have gotten safer either through in-mirrors or utilities. Yes, we do. Yeah, yeah. So I have with me Steve Monaghan, who oversees our safety workers' compensation safety division. We do have that information. We don't have it on hand with us, but we would be happy to provide that. But we do have data that's showing that the trend's coming down. So it's not just the administrative in-mirrors. Steve, Vermont employees are actually safer. Absolutely. We have data that shows that, for sure. And if we can continue that trend, hopefully everybody will jump on board and continue to help us bring those costs down. Other questions? Workers' compensation. I don't know if I'm going to go. The good news is it's fun. I know. I know. Yeah, they've worked really hard to provide for this. And I think our Vermont employers and employees have done the same. And I think that's really important to emphasize. So with that, I'm sure we're going to move on to some other questions, and you're free to move along if you'd like. Or you can stay right there if you'd like. Governor, your administration got a letter from the EPA concerning water quality. And it said, it's important that the state establish a long-term revenue source, since this is critical to successful and full implementation of the TMDL. So what's your plan for establishing a long-term revenue source? Well, again, I think you've read the rest of the letter. I mean, I know that's one point they made, but I think it was a fairly glowing type of letter saying that we're doing a lot of good things in Vermont. And in fact, they give us an extension in another area. I'm committed, as I've said, multiple times committed to water quality. But the details matter. We put forth, we have a plan put together that will a short-term plan that puts us through the next couple of years. I still believe that growing the economy, making Vermont more affordable, and growing revenues organically are going to give us the money that we need, the resources needed for continued water quality issues. But I don't think the knee-jerk reaction should just necessarily be to just add another fee. Maybe there's a way to reduce spending in other areas and incorporate another fee structure in itself or another tax. So I'm fully committed to water quality issues, including up our lakes and streams. But I don't believe that we have to put forth a fee at this point in time. Do you do it in the future? Well, possibly, but as I said, I believe that we have the means to look what we've done this year. Last year, we had a budget that didn't raise taxes and fees. I'm convinced we can do it again this year. And we, at the same time, are spending $82 million more in our budget through organic growth in revenue. And I believe that we can do the same. I also believe that we are not finished with the Massachusetts agreement through the conduit of renewable power through Lake Champlain. And I think they'll eventually come back to us, and if not them, someone else. So that's another source of funding that I believe that we can use for Lake Lena. So what do you know about the power line that makes you say that? Well, they've, Northern Pass has done, as you, I believe, reported on channel three. So I believe that's off the table. I know there's another one that they have considered, but they don't have all their permits lined up. And we do. I mean, that's what I continue to tell Governor Baker. Sometime, you'll be back to us because we have all the permits in place. You don't have any knowledge. I don't have any knowledge that would tell me. You're still looking at that main project, right? That's right. That's right. You know what you're gonna sign on as the gun legislation and whether you'll do that? Well, again, technically, we don't have that bill or 221 or 422 at this point. My hope is that we will have those in the very near future. And then we'll have five days to move forward. My plan would be, if everything goes right, to have all three bills and have one signing of all three. Not be a public signing? Yes. Senator Ash is considering a tax on copia, the description of copia, to raise some money to repay for treatment and addiction as well. Does that concept make any sense to you? Well, I haven't seen any details. I've heard rumor of this and read about it through some of your sources. But I haven't seen any details about what they plan to do with the money that they're raising. As you may recall, a week or two ago, we had a bit of a press conference announcing a $28 million settlement with tobacco. 14 million of that is going to go to opioids. So maybe some of that money could go towards the initiative that they're trying to fulfill. As well, the federal government passed a budget that has included money for opioids as well. And we're not sure about the detail. So what I'd like to do is find out what they want to do with the money first, the goals. Maybe we can agree on the goals and find a source of funding that we already have. But it sounds like you don't really think it's necessary. Well, the opioid crisis is something that is on the front of the site. It seems like a significant tax. Oh, yeah. Well, I just don't know. I don't know. I just haven't seen any details about how much this would raise or what their plans are with the funds. And obviously, we don't want to just raise a tax for no reason. We want to make sure that there's a viable plan to move forward. But as I remember, we have 14 million that we plan to use towards opioids. I think the point would be to have an ongoing source so that you could actually establish programs rather than the 14 million is going to stop in a year or two or the houses proposed before. So that makes it hard to set up something that might have to go away in a few years. Again, we've been spending a great deal of resources at this point in time using Medicaid, Medicare, and as well with some of the money from the federal government that I'm not quite sure and I'm not clear as to where that money can go or be utilized for. And for the federal government to give us more latitude, more flexibility is going to be important. We testified on that fact just about a month ago. And I believe that Congress will react to that because we're not the only state. We're just a little bit further ahead than most. But at the same time, I believe that we've been doing a good job working together and trying to solve this crisis. How do you feel about sending Vermont National Guard troops to the Mexican border? There's been no requests for us to send any National Guard troops anywhere at this point to the border with Mexico. How do you feel about the concept? I mean, if you were a pastor, is that something that you would comply with? Well, we would consider anything, obviously. That's what we do. If there's a crisis somewhere, we would consider that request. I've watched this issue. And it appears that the four states that have been requested their help are right on the border. So I don't anticipate us being asked to commit any troops. I'm not sure that that would be something that we would be eager to respond to. You mentioned crisis. Do you think this on the border is a crisis? Well, they know better than I do. Obviously, we're fairly north of the Mexico border. And the other states are living and breathing that. So it's for them to determine. So if you said it's not something that you would be eager to respond to? Would not be eager to respond to sending our National Guard to the Mexico border. And why is that? Well, again, I think it's outside of our realm, our geographical area, when there are other National Guard troops that are much closer than we are. So you haven't been contacted by the Department of Mass or all that security or anybody else? We have not received. You said that there are coordinate encounters. Yeah, we have not received any requests from the federal government on any deployment in regards to the Mexico border. But back to the guns, do you think anything is needed beyond 55, 221 of H-402? In terms of the violence issue, the underlying violence issue, yes. I don't believe that it's necessarily the gun, that this is an issue that we're all facing. And it's going to take a lot of additional steps, which I'll highlight in my remarks at the signing. The Senate Judiciary Committee is talking about school resources officers, armed school resource officers. What, I know this may have spoken out before, but what would you ideally like to talk about? Well, again, in my action plan, after the averted horrific incident in Fairhaven and after Parkland, part of my action plan was to do an assessment on the schools that's been completed. I'm looking forward to those results. I should have them in the next week or so to determine where we go from here. And the legislature has been very cooperative in carving out some funding. $4 million on the capital bill. We'll find some other resources for the additional $1 million. So we'll have $5 million to do something sooner rather than later in terms of our schools. At that point, after we get through these signings, I committed, as part of my action plan, to providing for a task force that is going to look at these various issues so that we can move forward and determine other ways to keep our communities and our schools safe. Resource officers, it could be part of that. There will be a broad group of about 15 members with different perspectives. And we'll take a look at this. Russell Barb is his own deadline for the work for a sexual planning plan. Do you have any plans of doing this further? Or are you willing to sort of let it drop? Well, again, these are serious allegations that were made. I look forward to it doesn't mean that he isn't going to provide them, but we look forward to seeing what he has so we can determine whether to move forward with this or not. Would you make any efforts on your own volition to find out whether it's passive? Again, I don't think it's being passive. I think we look back and there was an internal investigation completed. It's a little specific trick, but not to sort of. Again, I don't know what we would be searching for. Nothing was turned up in our internal investigation. Bar and Mr. Barr providing any details. I'm just not sure what we'd be looking for. Back to the school issue for a second. Are you at all concerned that schools are going to be these armed fortresses and how that impacts the students? Well, again, I'm concerned about that. But my primary concern is keeping our kids safe. So that from my standpoint, we need to do whatever we can to provide for the safety of our kids so that as I've said at other times, they shouldn't be afraid to go to school and our parents shouldn't be afraid to send their kids or put them on the bus to go to school. So we have to do whatever we can in the interim to make sure that our kids are safe and then start addressing this underlying issue whatever it is that's driving the violence. How important is it to you that the next educational secretary have expertise in educational management as opposed to other types of backgrounds that they might come with? Well, having an education background is important, obviously, but as I detailed, the State Board of Education had reached out and asked that we provide, that I provide some sort of parameters. What am I looking for in the next Secretary? And from my standpoint, as you've seen, the way I've built my team in the administration, Secretary and commissioners, they come from all walks of life with management experience. And I think that that's important, particularly with education. It's very complex. It's the single largest expenditure in state government, $1.7 billion. I think we need someone at the helm that has some management experience. But state law specifically says education. Absolutely. And for those who don't understand the process, I think it's important to note, it's the State Board that is going to be interviewing and taking applications, so to speak, and resumes, and then determining what three candidates they're sending to me. And then I'll choose from those three or send those three back and ask for more. So I think what they wanted to do, and I appreciate that, they just wanted an idea of what I was looking for. And management experience, I think, should be part of that package as well as education experience. Are you specifically looking for someone with, I guess, a non-traditional background for that role, someone who comes from the private sector, or? Well, again, it could be. We're just looking for it. I just wanted to make sure that we looked broadly, that we made sure that we're thinking outside the box because you never know. I mean, you could have, like our AHS Secretary, who's doing a terrific job, who's a restaurateur, have a Secretary of ACCB, who has a master's in education, was a police chief, and was involved in technology, and now he's in economic development. I think that, again, having the right people with the right attitude, with some management experience would lend itself well. But obviously, in this position, it's by statute, have to have some education background. But I don't want to preclude anyone from being moved forward, that clearly, when I'm looking for management as well. And you're led into the legislature a couple of weeks ago, outlining the bills that you had objections to. S-260, the water quality cleanup bill, EU-listed, new tax or fee, and also separation of powers. Senator Bray, whose committee wrote that bill, has been seeking an explanation for that from your administration, has gotten one so far. He consulted with the legislative council and they said there was no separation of powers issue in the bill. So I guess, is there an explanation for that? Do you or any of the other people who have come over here? Again, I appreciate their counsel and their advice to their senators. But from my perspective, when a bill prescribes that you're going to come up with a fee, that would tell me that they're telling the executive branch what to do. And that's not in their purview. If they want to come up with a fee of some sort, just do it. They have a legislature, just pass one. That's not what the bill does though. I mean, the bill- The bill calls for the establishment of a funding and ongoing funding source for water cleanup, which could be a new fee, in which case it would come before the legislature next year. Senator Bray said it could also be budget cuts. I don't think of what I saw. And I'm going to let these two fine secretaries answer some of this. But I didn't see anywhere where it said it could be. I saw where it said it is. But I'll let them react. So in particular, in section four, and I can hand this to you afterwards, part B3A3 directs the Clean Water Fund Board to determine how a parcel fee or other fee shall be assessed to property owners in a manner that corresponds to the effect of their property on water quality. It may be that that's a Scribner's error under a heading with the flexibility you describe, but that's certainly not the way it's written in the bill as past the Senate. And we flagged that as a concern. This is a copy of the testimony I provided to House Fish Wildlife and Natural Resources yesterday, outlining those concerns, and also sent a copy yesterday morning to Senator Bray. So he has them in his possession as well. I think it's also important to note that S260 has left the Senate. It's gone out of committee. It's gone out of the chamber and it's now over in the House. And now the House is dealing with it. So we're focusing on the House. So, I mean, Senator Bray's point is that he believes that since the Senate passed this bill unanimously on a roll call, both the Senate deserves an explanation separate from what's given to the House. And I guess we won't see it that way. I can speak to that, John. And I saw your earlier email as well questioning why you were not speaking publicly about, the question was why weren't we speaking publicly about our concerns with the bill. I do want to clarify the record. S260 was started in Senate Natural Resources. Both Secretaries, Secretary Moore and myself and many resources of staff across state agencies testified on the bill introduced in Senate Natural Resources. It then passed out of that committee and was committed to Senate agriculture. And that was the first time where we had an opportunity to testify about our concerns of the Senate Natural Resources version of the bill. And we testified in the Senate about our concerns of separation of powers and the fee that was in the bill. We reached an agreement compromise with Senate agriculture. They introduced an amendment on the floor of the Senate that we were perfectly fine with and happy with proceeding on. And that amendment was pulled. The current iteration of S260 has been passed. We didn't get an opportunity. Neither Secretary Moore or I were asked about the final version. We are now in the House. Senator Moore testified yesterday on S260. She's provided written testimony that was given to Senator Bray yesterday morning at 9 o'clock in response to his request to me for our reason. He was told on Tuesday I would not be appearing in his committee until S260 was committed back into the Senate. So we've been speaking very publicly about our concerns with the bill. Thank you. But I just want to follow up. You said the bill to the RECC to do something. But the legislature does that. Any number of times a year they order the court. So you have to complete all kinds of things. So how is this different? Directing for a tax or a fee is much, much different. They're asking you to come up with a concept of how it can be done. They're asking you to come up with a fee. That's what it says. And I would offer you to take a look. And just furthermore on that, if you look at the Senate passed version, it totally changes both the composition and the charge of the Clean Water Fund Board, which is made up of five cabinet members currently, whose charge is to allocate the Clean Water Fund across projects across state government. The further separation of powers issues in that bill is that it now has five cabinet secretaries reporting directly to the General Assembly on how to budget that money and completely turns the budgeting process and the constitutional process of the governor to present his business to the legislature on its head. Thank you, Queen Water. Governor, do you have any updates about the phosphorus innovation challenge thing? Is that something you can do? We do have a few days. Sure. So we've continued the Agriculture Agency, Agency of Commerce, Community Development and Agency of Natural Resources have continued to work together to develop what we've termed a reverse pitch. We expect to release it within the next two weeks. That initial solicitation will give folks approximately two months to submit their ideas since we first talked about the phosphorus innovation challenge. Secretary Shirling, Secretary Tevitz and I have been out doing informational interviews with project developers in Vermont to understand what technologies are sort of under development here, where those opportunities exist and we're really excited about the prospects. Our hope would be to announce the stage one award, so those successful applicants later this summer and move quickly towards the prototyping and ultimately the construction of these technologies. As you may have seen, there was a report from UVM earlier this week that specifically identified the phosphorus imbalance in our watershed as one of the challenges facing us when it comes to water quality, making the phosphorus innovation challenge even more timely. Can you remind me, does the legislature need to sign off any funding for that program? Not for stage one. Stage one, we have funding to cover those costs. If we get to the stage of actually standing up these projects where we may want to make a capital investment, we'll absolutely need to come back to the legislature for that conversation. That hasn't been addressed. It has not. You had a meeting yesterday with Richard Piebie of Coverage Co. Any comment, reaction to that and where does that issue stand in your mind? What do you want to see happen? Yeah, Coverage Co continues to be an issue that we have great concern about and the loss of cell service broadband for those in the southern most part of the state. It was a good meeting, had learned a few more details. They have some resemblance of a plan that they'd like to, there's details in the plan that would have to have considerable work done in order to complete that. But as well, on the other side, and I'm not, I can't announce anything at this point, but there are two others who are working on this as well to other private companies that are going to help us out in the interim. So we should have some details on that the next two to three days. I think they were supposed to testify today at the far house committee that came in at close time just to write her last. Coverage Co. Well, I think it's somewhat unfounded in some respects. This is utilizing a process as being utilized by most every other New England state and right across the country, as well as even teachers. So this isn't, I think the state employees have utilized another approach over the last few years this is just bringing it into the same approach that we're taking or others are taking throughout New York. So I don't believe that it provides any undue suffering by any of our state employees. Do you think it will cost employees? I don't know that it will. In fact, I think it's going to cost less in the long run with reduced costs for health care. That's what we discussed this a lot in this building. How do we produce the cost of prescription drugs? And now we found a path forward to doing so. And I think it will have a benefit in the future by lowering the cost of health care throughout. So I failed to see where I'm not sure that I completely understand what the concern is. These are generic drugs that are replacing the ones that are stipulated. So one of the concerns that I've heard from the community and I have not verified it is that not all of the pollutants have a generic drug to match it. And that I think is where some of the major concerns coming from. You've got to bring it there. I mean, I know it says either a generic or a therapeutic alternative, but I think hearing therapeutic alternative doesn't give people much comfort knowing that there isn't a real drug to match what they're taking now. Commissioner, a few resources here and she can fill in them. Hi, I'm Beth Bastigie, commissioner of human resources. And with respect to that question, there would be a process that employees would go to to work with the pharmaceutical company and their doctor for an appeal. And that's a practice that's also happens nationwide. And overall with our current pharmacy benefits manager about 70% of those appeals are actually approved. And the other ones most likely are that they move on to an alternative and that works for the employee. Diabetic test strips? So diabetic test strips would obviously be available to all employees. A particular brand might not be available. So while we might be not providing the employee Kleenex, you might provide them first opportunity would be the Hannafer brand. But if that doesn't work for them, then be a puffs, you know. So it's kind of the same alternative. So employees would obviously be able to get diabetic test strips. I mean, the list that I saw just didn't say but brand of diabetic test strips just said that test strips and the machine goes with it. Yeah. And that list is a standard list that was provided to the VSEA upon their request. It's not actually the final list that would be that will be the list January 1st when that plan goes into effect and people with diabetes will be able to have test strips and test that and test it for whatever it used to be used for. Governor, the house is now considering the minimum wage that was afforded. Is that a non-starter for you? Is that something that will be coming? I haven't seen anything that's improved the bill. I still have significant concerns about the overall effects on our economy. I think it will have a detrimental effect on our economy, particularly in the rural sections of Vermont and through on the Eastern border of our state where we are day in and day out, they struggle to compete with the Hampshire. This will not help. So from my standpoint, I want to see more for mothers make more money. I think that's important, but I think as well that the supply and demand will provide for that and that this is just artificially raising the cost of living for Vermonics. Will you be feeling it? I would say that if it doesn't make substantial changes between now and then, it could see a veto. So are you willing to consider some increase and not what they've done right now? No, I'm saying, I think ratcheting up this and I've been part of this. I know I've talked about this in our press conferences, but I remember when I was in the Senate and we had this discussion, this debate and we had the cost of living approved and some of the debate was, we'll never have to go through this ever again if we pass this bill. Well, we've gone through it once since and now we're going through it again. So I believe I'm very concerned about the small businesses of the state, particularly the mom and pops and there are many. I think the figure I saw was 80% of the businesses in Vermont are under 25 employees. So this will have a detrimental effect on the businesses as well as the overall economy of the state. What would the substantial change mean? I mean, put it back on the shelf. That would be a substantial change. And what are you saying? I would take that. Again, what's a real change? This is my idea. And I believe that if we focus on growing the economy, making Vermont more affordable, that we'll have organic wage growth. And that will be a positive aspect of doing so. So let's focus on how we can make Vermont more efficient. And let's look for cost savings throughout government so we don't have to raise taxes and fees. And we'll see wage growth. It's been a long time since we've had, I mean through good times and bad over the last, I don't know what it is, 20, 30 years, there's been the lower, the working and middle classes have lost purchasing power. What makes you think that the economy will suddenly step up and start paying people living wage? Well, some of it is in regards to the demographics of our state. We are losing our workforce. We have fewer people working today than we did seven years ago. And when that happens, and you can see it at job fairs, there's a lack of people in Vermont, lack of workers. So when that happens, it's a supply and demand issue. And so you have to offer more money in order to get the employees that you need. But that doesn't seem to be happening. I believe it is happening. Can you conceive of a change substantial or not? I haven't seen one yet, but we'll see what they come up with. Thank you very much, appreciate you coming in.


Welcome to Gastrointestinal System Anatomy.  How does the food you eat turn into energy for your body? Let's find out. We'll explore the main parts of the gastrointestinal system and each part's function to answer these questions. The abdomen is divided into four quadrants. These are the right upper quadrant, the left upper quadrant, the right lower quadrant, and the left lower quadrant. Now, let's take a closer look at what's in each quadrant and how each organ helps your body break down food into the water, minerals, and nutrients your body needs. The liver is a large organ. It is found in both the right upper quadrant and the left upper quadrant. The liver performs many functions to keep the body healthy, but its primary role in the digestive system is to process and filter anything you eat or drink. The gallbladder is a pouch that's under the liver in the right upper quadrant. It works with the liver to both store and concentrate bile from the liver, which breaks down fats. The common bile duct is tube-like and located in the right upper quadrant. It carries bile from the liver and gallbladder into the duodenum by passing it through the pancreas. The stomach is located between the esophagus and small intestine in the left upper quadrant. This muscular organ holds food and mixes it with stomach enzymes. This breaks down food so it can be absorbed into the body. The cecum is found at the first part of the large intestines. It connects the small intestines to the colon. Located in the right lower quadrant, the cecum further digests food from the small intestines. The appendix is located in the RLQ, just below the cecum. We don't know its exact purpose, but we do know we can live without one, if necessary. The ascending colon is located between the cecum and the transverse colon in both the RUQ and the RLQ. Also called the right colon, the ascending colon is the first part of the colon. It accepts partially digested food from the small intestines. The transverse colon is the horizontal piece of colon that transverses from right to left between the ascending and descending colon. This is the longest part of the colon, and it's located in both the RUQ and the LUQ. The transverse colon plays an important role in digestion by absorbing water, minerals, and nutrients from our food. The pancreas is located behind the stomach in the RUQ. Pancreas enzymes aid in the digestive process by breaking down sugars, fats, and starches. The descending colon is part of the large intestines, located just before the sigmoid colon in the left upper quadrant and the left lower quadrant. This part of the colon stores feces to be emptied into the rectum. The sigmoid colon is part of the large intestines. It's found in the left lower quadrant. This is the last portion of the colon, and it attaches to the rectum. The sigmoid colon is tasked with holding feces until you go to the bathroom. The rectum is located between the sigmoid colon and the anus. The rectum receives stool from the colon and holds it until you can go to the bathroom. Every organ in your gastrointestinal tract works together to help your body break down and process the food you eat. They turn food into water, minerals, and nutrients necessary for your survival. In this video, we explored where each organ is located and how they work together. This completes Gastrointestinal System Anatomy. If you enjoyed this video, please subscribe to our channel. We'll alert you when we have something new.


Good morning everyone and welcome to the December edition of our research showcase. This is Dario from the Wikimedia Foundation research team and today I'm joined by Leila Zia, senior research scientist in the team and by a number of co-authors who are here in the Hangout today to talk about work that we've been doing at the Foundation in collaboration with researchers at APFL and Stanford University on using recommender systems to fill knowledge gaps in Wikipedia. I'm not going to spend too much time giving introduction. Leila's going to go over this. Just a few service announcements. As usual, we have our IRC channel. Jonathan Morgan is going to be hosting on IRC. So if you have questions, please raise them to Jonathan who will ask them at the end of the presentation. And today we only have one presentation. So we'll have ample time for Q&A at the end of the session and stick around and enjoy the show. And with that Leila, let's say it is yours. Thank you Dario. Hi everyone. My name is Leila Zia. I'll be talking about recommendation systems and knowledge gaps in Wikipedia today. This is a joint work with a long list of people, but specifically in research. It's with Bob West at APFL, Tiziano Picardi, Diego Saez at the research team at Wikimedia Foundation and Michele Katasta and Eurydeskovitz at Stanford. It's been a while since we have talked about recommendation systems and knowledge gaps in Wikipedia. I figured it's a good time to step back now that we have the full hour and talk a little bit about how we have started and where we are today and where we want to head next. I think of this presentation as usual as a kind of presentation that hopefully it will engage more conversation and it will create more questions than it answers. So with this, let's rewind. Let's go back to 2015. Back then, we started looking at these maps of Wikipedia. So basically, this is the map of the world and every point on the map is associated with a geo-coordinate associated with that map, with that point on the map. If the point is white, that means there are 10 or more Wikipedia articles associated with that geo-coordinate in Wikipedia, in English Wikipedia in this case. If the point is orange, there is one. And if it's orange, it's fewer. I can't remember how many. It's logarithmic scale. If it's blue, there is one. And if it's black, it means there is no article about that geo-coordinate on Wikipedia, at least as much as Wikipedia knows it. What you see here is that if you speak English only as the language that you speak, there are some places in the world that you can learn about if you read Wikipedia. Specifically, good parts of US and western part of Europe, some places around Iran for some reason, and then sporadic dots all across the world. Now, as you go to other languages, you see that the number and intensity of dots drop significantly, which means people in these languages have basically less access to content. If Russian is your native language and you want to learn about the world using Wikipedia in Russian, this is how much you can learn about the world. If you speak Spanish, you basically can learn about Spain, but the whole South America is very sporadically covered, not fully covered at all. And if you look at Portuguese, which is a language you expect to see more of in South America, definitely that doesn't happen there now. You can say that there are good chunks of South America where you have Brazil and there are basically the Amazon jungles, which are not heavily populated, but even in very populated areas, you see that the coverage is not very good. You basically have a best one article for most of the points. If you speak Arabic, the world becomes much darker. If this is your only language, this is how much you can learn. And interestingly, if you overlay all languages that we have a Wikipedia language edition for, this problem doesn't go away. So basically, if you speak all the 300 plus languages for which we have a Wikipedia language edition, this is how much you can learn from Wikipedia if you want to learn about the world. Now, by looking at these plots, you can gain a few insights. One is that we have a lot of missing articles across languages. The second thing is that not all of these missing articles are available in at least one Wikipedia language. As you saw in the overlay of all languages, there are still a lot of points that are dark or don't have so many white or orange points associated with them. This means that Wikipedia alone won't be the solution for all knowledge gaps. We need to look outside of Wikipedia to try to close the gaps that we have in Wikipedia on top of trying to look across languages and try to import content from one language to another. So our goal in general, the goal we set in 2015, which is to date relevant, is that we want to increase article coverage in terms of the number of articles in different languages and the contents of these articles within a language by identifying and prioritizing missing content and routing attention where it's needed. So we started a work back then, growing Wikipedia across languages by article creation recommendation. This was a joint work with Ellery Vulkshin, Robert West and Eurideskovitz. What we did back then is that we defined a system. This is what the system would do. The system would find articles that are missing in a given language, let's say French Wikipedia, and then it would prioritize them with some notion of importance. The notion of importance that we defined in 2015 was that we built a prediction model that would predict how many page views the article would receive if the article gets created in French Wikipedia as an example of a language. So we find missing articles in a language. We rank them according to some notion of importance. And then what we do is that we recommend these articles to editors of that language, of that Wikipedia language. We did an experiment back in 2015 where we chose the top 300,000 articles that were missing in French Wikipedia. We divided these articles into three sets of 100,000 articles each. And then we looked at the 12,000 editors who had edited at least once in the past six months prior to the experiments on French Wikipedia. And what we did is that we used the first 100,000 articles to do personalized recommendations to the first batch of editors. To the second batch, we did random but important article recommendations. And we kept the last batch of 100,000 articles untouched. The goal of doing this would be to be able to say that if you don't touch the articles, if you don't recommend them, how much organic growth will happen in Wikipedia on these articles and then how much would be the impact of personalized recommendations or random recommendations. What we learned back then is that you can increase the growth in article creation rate by a factor of 3.2 if you do personalized recommendations versus doing nothing. And we also learned that personalization actually matters. If you do personalized recommendations versus recommendations that are from important articles but are not adapted to the user's interest, you will get basically better growth. The activation rate here is basically captured by the fact that when you receive the recommendation, whether you actually act on it or not. Whether you start editing the article or not. And we see twice more growth in those who had received personalized recommendations. On the ground, basically the result of the experiment meant that we started a gap finder project. So this is a tool that you can go to. You specify a source language and a destination language. And then you can there basically search for the content articles that are missing in one language that are available in another language. And then you can translate or create from scratch. We also built an API and that API is being used in content translation suggestions feature. And this API is perhaps used by other people who are building tools. This is not so much something that we know much details about today. So here's how a gap finder would work. You choose your language and say English and the destination language Persian. The only reason that I'm showing you English here is that I want you to be able to read the recommendations. Obviously you can do it from Persian to English and in any direction in both directions for other languages. You specify what you're interested in, in my case, agriculture. And what I see is the articles that are available in English but missing in Persian with some basically incentives for creating these articles. So these are the page views that the article has in English Wikipedia. Once you click on one of the recommendations, you basically see the full article in English Wikipedia and you can decide whether you want to create the article from scratch, translate it or flag the article. Whether it's not interesting for you or it's not notable in your language. You can do this obviously for other languages as I mentioned. So for example, this is an example from English to Malagasy. You type climate change if you're interested in that topic and you can see the list of articles that don't exist in Malagasy but they exist in English and you can translate or create from scratch in that language. So now what we learned in 2015 was a few things. One is that recommendations work and personalized recommendations work. But from 2015 to now, basically starting 2016, we went through what I call a series of revelations. First, we learned about stubs. Not that we didn't know about them but we started hearing more and more about the issues around stubs. We were reminded by the editors that there are many articles already in Wikipedia that are stubs and there are many comments about this topic all across the different lists that we are part of. I'm listing one of the examples here, Stuart says on Weekly Research L, we have enough low quality sub articles that need human effort to improve and we are not really interested in more unless either they demonstrably combat some of the systematic biases we are struggling with or they demonstrably attract new cohorts of users that can do the improvements. We also learned about onboarding newcomers and the challenges of onboarding newcomers. We learned about initiatives such as the Africa de-stubbathon project with the goal of de-stubbing the articles about Africa in different languages. We had user interviews basically with people who organized these kind of de-stubbathons in Africa. What we learned through these user interviews is that a lot of manual work is going into building templates. As somebody who organizes an editathon in a large scale, basically you're embarking on a project where you want to engage people who can easily, by reading a set of steps, learn how to edit Wikipedia, basically you're challenged to create a set of templates, a set of step-by-step instructions that can help the user understand how to edit Wikipedia. Specifically in the case of subs, one of the biggest challenges was for the editathon organizers to basically extract the information that a newbie needs to know on how to expand, for example, an article on biographies. For example, if this is your first time coming to Wikipedia and you want to expand an article on biography of Person X, you need to know how to do this and there are some basic things you need to know. For example, what sections should go to a biography article? What should be the length of a section? What kind of other articles should the section link to? What kind of images do you usually put there? These kind of basic information that experienced editors have learned over time are completely unknown to newcomers and this requires creating these templates from scratch requires a lot of manual work from editathon organizers. I think this was something that we empathize with very heavily at the personal level, especially as you're going to regions where volunteer resources are extremely limited. They're always limited, but there are regions where we have very much like clear lack of resources. If you have one volunteer whose time is spent on creating templates from scratch, going across different languages, this is like a big challenge for the work that this person does for the movement. We also looked at existing initiatives. There are initiatives such as Wikidaheim or Macamoon in French Wikipedia where these initiatives try to basically do the work of editathon organizers in a more systematic way. What they do is that they focus on the specific types of articles and then for these types, they extract templates of sections and characteristics of these sections. They hard-code these in the back end, basically. As the user, when you go to the websites for these projects, what you see is that if you want to expand an article about a city in France, you will get recommendations on the right-hand side that tells you these sections are missing or these sections are incomplete. It also gives you some more contextual information. Sometimes, for example, on average, how many characters or bytes are available or how many characters or bytes are associated with such a section in similar articles in French Wikipedia. What is nice about these initiatives is that there is a confirmation that there is a need for providing step-by-step guidelines for newcomers or less experienced editors to learn how to expand already existing articles. It also provides an opportunity for us to learn how we can surface the work that we do if we do research in this space. We also lastly looked at numbers. This is where we learned that in English Wikipedia, 37% of the articles have a stop template. I'm talking about the template because there are always discussions about whether the template has not been removed or has been removed. Obviously, this number doesn't tell you the exact count of stops. But it gives you a sense of the rough numbers that we're talking about. Out of the more than 5 million articles, 37% of them are likely to be too short to be of encyclopedic value. Only 1% of English Wikipedia articles have been labeled as good or better. And 80% of the sections created in English Wikipedia are used only in one article. We have a potential because every month 14,000 accounts get created across Wikipedia projects. And the other thing that we did when we looked at the numbers is that we looked at plots like this. So what you see here on the x-axis is the section counts. So you see section count from 0, 1, 2 all the way to 14. And on the y-axis, you basically see the normalized frequency of articles that have such section counts. The orange plot, basically this one that I'm walking with the mouse on, this is plotted for all articles. This means that more than 20% of all articles on English Wikipedia have only one section. Now if you look at the blue plot, the one that I'm walking on with the mouse now, what you see is that pretty much 0% of them have only one section. In fact, close to 20% of them have six sections. So this is telling us that for high quality articles on English Wikipedia, looking at sections would be one of the right places to start from. We know that articles in general have very few sections, but the high quality articles have more sections. So that was a signal for us that maybe we can start by looking at sections and figuring out how to do section recommendations. So we had a few takeaways here. One is that we have a lot of missing content in Wikipedia articles. And the other one is that edit on organizers are spending a lot of time for manually extracting templates for onboarding newcomers. There are initiatives that have identified these needs and are working. But the problem for these initiatives is that they cannot scale. Basically the problem is that they can only focus on specific topics because manually creating templates for many topics is not scalable. So we started imagining and this phase of exploration and imagining was heavily, we had heavy interactions with one of our designers who has been with us on this gaps project for many years now. We said, okay, let's step back and imagine. Let's imagine that we have a system that breaks down the article structure into different pieces. For example, you can imagine that you can break down a Wikipedia article to lead paragraph, info box, sections, images, links, references, etc. Suppose that the system can provide you with recommendations on what's missing from an article and it can tell you that this missing content should be added to it. Also imagine that the system can provide you more in-depth information about the thing that is missing. And this is by the way a key factor for all the recommendation systems that we're working on these days. Basically we need to know why a recommendation is given and what kind of other in-depth information can be provided to the editor so that the editor can make the right choice. The in-depth information, one example of it, for example, can be that you can say early life section in biographies on English Wikipedia usually has on average N characters. So we decided to start with sections and building a section recommendation system. Before I move on to some details about this section recommendation system, I want to wait a little bit and maybe take a few questions if something is unclear because I've been basically moving quite quickly on this part. John, is there anything from the room? Nothing from my RC at the moment. Okay, I'll move forward then. Okay, so this is a joint work with Tiziano Picardini, Michele Katasta, Diego Saez, and Robert West. So here is where the problem starts. Suppose you have an article, this is English Wikipedia article about a city, Kurdish city in Iran, which is called Sanandaj. You look at the article and you see that there are four sections in this article. Now two of the sections are already kind of the infamous sections, references and external links that pretty much appear in all articles. So basically the article has two sections, society and famous people connected to Sanandaj. Now suppose I'm an editor and I want to expand the list of sections for Sanandaj and start working on it. If English Wikipedia is the only language that I speak, what I'll do is that I'll start looking for other articles that are similar to Sanandaj. In this case, probably I will look at the capital of Iran, Tehran. I know that's a city, so I will go and look at the list of sections and I say, okay, the capital has these sections, so probably another city in Iran should have at least a subset of these sections. This is what we call the intra-language information extraction. So the only thing I know is in English is English and I look at English Wikipedia articles to extract information about the section structure and the section content in an article with fewer sections. The other thing I can do is that if I speak more than one language, in this case I speak also Persian, I can go to Persian Wikipedia and I can look at the article on Persian Wikipedia. And what I learned is that these are the sections that are used in Persian Wikipedia. And in this case I actually have cross-checked them with English Wikipedia and I know that not all of these sections are the same. So I can also extract information from other languages when I'm creating an article on English Wikipedia. This is what we call the inter-language information extraction. For most of this presentation, I'll focus on the inter-language recommendations. The inter-language recommendations is something that we have recently started with Diego working on and we can discuss them more in the discussion and Q&A part. So here's the problem statement. Given an article A in a given language L, we want to recommend the list of sections to be added to the article, considering similar articles to article A in the same language, language L. You see on the right-hand side the design recommendation by POW. If you have such recommendations, you can basically list them potentially on the right-hand side of the article for a left-to-right language on visual editor and you can empower the user to basically edit the article by adding new sections to the article. But the problem is that we don't know how exactly to define similarity. There are at least two ways for doing this. Looking at an article on English Wikipedia, there is the article texture that we can use, but there is also the categories that we can use, which usually show up in the bottom of the article on English Wikipedia. This is exactly what we're going to do. In the next two slides, I'm basically going to show you nine months worth of work in three minutes. So the first set of approaches that we tried is article-based recommendations. Here we have tried two different approaches. One is that suppose you start, sorry, let me step back. For all these systems, we basically have an input, which is the article. In this case, for example, Stanford, California, think about it as basically the city, not the university. So this is the input article. And at the output, what we want to receive is a list of sections to be added to the input article. So what we want to see in the output is a list of sections that are missing from Stanford, California, and the recommender system should tell us to add to the article. So one way to do this is to look at the article text. So we look at the text or content of the article, and we build a topical model for each article using the text, the full text of the article. This way, you will get basically a matching or a mapping from each article to, let's say, 200 topics that would be provided by an LDA model. And then what we do is that through a series of steps, we count how many sections are available, how many times a section has occurred in a given article, and then we normalize those by the LDA weights of the article belonging to a topic. And through this topic modeling, what we do is that we create a list of recommended sections for this article. Another approach that we can try is that we can do matrix factorization. So here we can say that, let's forget about the text of the article, because some articles, although their text is similar to each other, they're not actually similar in terms of structure to each other. So what we can do is that we can ignore the text of the article, and instead we can focus on the sections of the article. So here what you see in this matrix on the right-hand side is that you see the list of articles in English Wikipedia. So assume that basically you have 5 million plus rows in this matrix, and you have all the sections in English Wikipedia. So this is more than a million sections. And this matrix basically tells you whether section X or section J belongs to article I or not. So you create this matrix, and then you do matrix factorization, and you use collaborative filtering to get basically a set of recommendations for sections to be added. The gist of how this approach works is that generally collaborative filtering works in the context in which you have a user and you want to see what products to recommend to the user to buy. What you do is that you're going to find other users that are similar to this user. You look at what they have bought, and if many of them have bought a specific thing, you will recommend this to your user who is similar to these other users. In this case, basically user corresponds to an article, and this product corresponds to a section. We have an article, and we want to see what are the other articles that are similar to this article, and what sections do these articles include? Let's include those in article X that we are focusing on as well. There are advantages for using collaborative filtering, and that's one of the reasons that we wanted to use it and see whether it works well in our context or not. One of them is that it kind of respects the serendipity of Wikipedia better than some of the other approaches in the sense that it doesn't focus on the popularity measures of how many times a section has occurred only, but it also considers other factors, and that can help us basically find sections that are not very frequent but should still exist in our recommendations. These are two approaches, and I'll come back to the evaluation of the approaches after I tell you the next one. We also tried four other approaches using the category network in Wikipedia. In total, we have tried around seven different approaches to see which one works better. Here, we basically ignore the text of the article, and what we do instead is that we focus on the category network of Wikipedia. We look at Stanford, California, and then we look at the list of categories that this article belongs to. The first thing that we do is that basically the idea here is that we want to use the category network of Wikipedia to find similar articles. The idea would be that if all articles that belong to a category should be similar to each other in some sense. The problem, while in theory this is correct, specifically for Wikipedia category network, this is not correct unless you prune the category network. The reason is that you can have categories that many things belong to. These may not be, first of all, hierarchical categories, but they also may not be textonomical, even subsets of them. We need to prune the category network to make that into a textonomical category network. Now, I am almost proud to say that we have failed in the general task of pruning the category network for Wikipedia. However, we have developed a methodology for pruning the network specifically for the task of section recommendations, which we can cover in the following research showcases. It needs a little bit more time to dig deeper in it and see how we have done it. But anyway, we pruned the category network and the idea here is that if a category is not pure, we basically throw out that category. And if it is pure, we include that and we look at all articles that belong in that category. Now, one of the easiest things you can do is that you know Stanford, California belongs, let's say, to university towns in the United States. That's category three. So you look at this category three and you look at all articles that belong to this category three, and then you look at the sections in these all these articles. And then what you see is that, for example, 79 times education has occurred as a section in these articles, or 19 times history has occurred as a section in these articles. So you basically can learn how many times each section has occurred, and you can also learn about the core currents of sections within these categories. Now, once you do this count base, you count the sections in these categories, you can basically directly create a ranking of these categories and recommend them. That's one approach. Basically do nothing else other than counting. The other approach would be, again, to use collaborative filtering in this space and see what our collaborative filtering can improve the results. We can also do collaborative filtering and then do learn to rank, and that's another approach that we may be able to improve the results with. And lastly, what we can do is that we can use the count based approach and do learn to rank and then see what the results will look like. One thing I want to mention here is that learn to rank in general is useful in the sense that we know many articles belong to multiple categories. So for example, let's say Stanford, California in this case belongs to two categories, C1 and C3, two pure categories. What learn to rank allows you to do is to merge the recommendations by these two categories instead of focusing only on one of them. And we want to see if by merging the results, we can get better recommendations than by just focusing on one of the two categories. Intuitively, you expect that if you basically combine information, you can reduce the noise and then you can do better. If the way you have cured the category network has done a good job. Okay, I'll talk briefly about evaluation. So there are seven approaches and we have done two types of evaluation. One of them is automatic evaluation and the other one is human evaluation. In the automatic evaluation, what we have done is that we have created a set of test articles and then on those we have computed precision and recall. So that's automatic. And the other thing is human evaluation where we have done crowdsourcing for gathering labels on whether a recommended section is a good section for an article and we did this both with experienced editors and also using a crowdsourcing platform. I'm not going to show you all the evaluations for all the different seven approaches. They are on the meta page that was linked earlier and will be linked at the end of this presentation. So what you see here is basically this blue line here that I'm walking on with the mouse. That's the precision for the count-based approach using the category network on Wikipedia. And this is the recall, this other blue one. You see the green one, which is basically slightly over it, is what we gain by using learn to rank on top of just using the raw counts of section occurrences in the category network. And you also see basically the upper bound on recall and precision with this red plots. What you see here is that we are doing a reasonable job in terms of precision recall in the blue plots, but generally they're kind of low. This is not the kind of thing that you want to put in front of a user. Now, the issue that we suspect can be the reason for this relatively low precision and recall is that we are doing exact matching of section titles to the already existing sections in a Wikipedia article. And the problem is there are two problems. Sometimes there are typos in sections. And if it's not an exact match, our algorithm basically considers that as a bad recommendation. But there are also sometimes semantically related words that are being used interchangeably, for example, a movie or a film. These both can be sections and our algorithm at this point will differentiate between these two while a human will understand that these two are pretty much the same thing. So we expected that if we do human evaluation, we get better results. And that's in fact what we see here. So what you see on the right hand side is that there are two types of evaluations. The editor evaluation, which is this is experience editor evaluation, which is kind of the lower blue scion plots that you see. And then you see the purple one, which is crowd-based evaluation, which is higher. So a few things to note here. One is that obviously a crowd of non-recommendians is more linear. Let's say with close to 97% here, 98%. Wikimedians are around 90% here. But generally what we see is that outside crowd is more lenient, which is expected. We also see that generally the method is doing really well. So basically when we put this in front of the comedians, the recommendations seem to be reasonable. If we do only one section recommendation, the precision and recall is around 90%. And then if we do as we increase the number of sections that we recommend, the precision and recall drop. And when we are recommending around 10 sections, that's around a little bit more like 78%, 79%. So we are obviously happy with this result. It's much more than what we expected. And it's also in our context is not really necessary to make the system completely perfect in terms of section recommendations. Because you always will have the human in the loop, whether this human is going to be the person who is going to be coding Macamoon. And there are going to be 1,000 sections recommended to a series of articles that an experienced editor can go to and accept or reject. Or whether the human is a newcomer who can basically evaluate whether the quality of a section is good or not. So that's where we are with evaluation right now. So let me go to conclusion. For this specific second part of the research, we have introduced basically the section recommendation problem. We have stated it as a problem and we have defined what it means. We have explored several methods using features derived from the raw input article. These are the topic based methods as well as Wikipedia's category network. We showed that the category centric count base approach does the best. Of course, if you do learn to rank, you can do a little bit better for a Wikipedia language with the large category network such as English Wikipedia. We also have learned, and this is I think one of the best learnings like for me is that English Wikipedia's category network is key in offering useful recommendations. And it's important to call it out because we generally complain a lot about category networks in Wikipedia. At least like whenever we sit as researchers and we want to work with category network, it's a very hairy network that it's very hard to work with. But this is the first time that we are seeing that if you actually use the category network in smart ways, it has a lot of signal in it that can make recommendations, interpretable recommendations possible, which is key in our environment. Now I'm going to go to discussions and then open it up basically for questions. These are a few things we have been thinking about. So one question is, what about non Ian wiki, our usual number one question. So the reality is that the methods that we have developed are not language independent, we can at any time apply these methods to any language. The caveat is that if your language is very small, or the category network in that language is small, the category based approach will not work. And we have seen that basically the LDA and topic based approaches won't work. So if the language is small, we need to think about something else. We since we expect that the methods work in other languages in order at least in other large languages. The next thing we want to do is that we want to try the method in French Wikipedia. And that's something we're working on right now, basically adapting the methods for French Wikipedia, building recommendations and running it by French Wikipedia editors to evaluate the quality of these recommendations. And if the language is really small, that's the part that we have started working on right now. So the idea here is that if your language is very small, we need to look at other languages to extract information. This is something that actually is being done by editors on organizers. So they basically use other languages that are similar to that language to create templates. So now we want the machine to do this and it's not an easy task. Some of you who are on Wikipedia Research L have seen the emails by us asking about we're basically stuck with a problem of section alignment. We need to be able to align sections across languages to be able to approach this problem. And this is something Diego has been spending quite some time on and would be happy to discuss with you later. We also need to improve the section recommendations. In general, although the result is good, there is room for improvement. So basically one of the easiest things we can do is that we can try to source signal from other languages, the inter-language part. Even for English Wikipedia, if we can source signal from a language like Persian Wikipedia, there's probably some things for us to learn there from a different language in even a big language. The other thing which is kind of a more immediate low-hanging fruit is the semantically related sections. So right now the algorithm is not differentiating between these. And it's not very hard to improve the work of the algorithm, at least in the languages where semantic analysis is being done relatively smoothly to basically combine the sections that are semantically related. Providing more in-depth information to the editor is something that we are not doing right now what we would want to do. Basically, other than saying this is the section that should be added, we want it or that you should consider to add, we want to be able to say what are the characteristics of such a section. This should be an easier step than all the things we have been working on so far. The other thing that we are not talking about right now is the order of the sections. So suppose you're an editor and we tell you here are the sections you should add. How should you know what is the order of these sections? This is not something that we have worked on yet and probably someone should pick this up in the future. It's not really immediate, but at some point we need to be able to address the ordering problem. Including the less frequent and long tail-up sections is something which is important. While we have learned through this research that there are very few unique sections on English Wikipedia. Let's say there are around, if I'm not wrong, around 200,000 sections that are not unique and 1.6 million sections in total. This means you have around 1.4 million sections that have been used only once. This long tail is quite long and many of them should not really be used potentially, but in this long tail there are also a lot of valuable information. The question is what are the valuable sections that are being added by editors that our algorithms won't pick up right now because they are focusing on frequency and we need to be able to pick up. There are different ways this can be approached, but this is definitely one thing that we should come back to at some point. The other thing is the problem of entry point. This is actually something which in this research basically we consider the article as an entry point, but there is no reason to not think about the category as the entry point. This is one of the strengths of the line of research that we did with the category network on Wikipedia. What we learned from the editors and organizers is that sometimes they just want to be able to give a category and get recommendations on what sections to be added to articles in that category versus giving the articles one by one and getting recommendations for sections to be added for the articles. The research is adaptive in that sense and can respond to this kind of entry point. What's next? I think the biggest thing which is next is more languages. We need to build a tool API for testing it further in other big languages, let's say French Wikipedia, but also in smaller languages. We need to think about the problem of basically continue thinking about and exploring what if the language is much smaller than French Wikipedia. We need to test in these smaller languages. We need to develop a different set of technologies for these smaller languages. Most probably the category approach won't work. There's a project I called earlier Macamoon in French Wikipedia. We have done some early conversations with the team behind Macamoon and we may want to start working with them to provide some recommendations to help them scale the scope of their work to beyond just the articles about cities and towns in France. And then at the end of the day, we need to put this in front of less experience or newcomer, less experienced editors and newcomers and see whether this kind of system actually helps them in practice. We can do as much evaluation as we want right now, but once the system is in front of the user, they will need to make the call of whether this is useful for them. And I want to call out that this last bullet point is a product heavy design heavy step. So we have tried to develop this research by continuously discussing it with the community and the designers in the foundation to make sure that we don't end up in a direction which is heavy and exciting in research but less exciting in terms of product development. But this is going to be a very important steps for product and design to get more involved if we want to put this in front of new editors in the long run. And then obviously other types of recommendations. So we have started discussing with Miriam how we can do image recommendations for articles. Again, this should technically be an easier task that we should be able to do a good job in and it will be quite impactful because images are basically the way we can communicate across languages without requiring any words. So can we bring these images to give you the articles and make them more useful and understandable by their corresponding audiences? Info box recommendations is another item which is a simpler problem than the one we had been working on. And probably other kinds of recommendations. I'll stop here and then we can talk. Thank you. Thanks so much, Lila. I'm going to open up the floor to the room for any question we may have from IRC. I've got two questions and one possible comment so far from IRC. First question is from computer MacGyver. They say first, great work. And then to what extent do the related articles from category or text recommendations align with Wiki projects? Could Wiki projects be useful input training data for any step? Yeah, that's a very good question. So this is something that we also discussed with Diego and Daria because Erin is also doing some work on with Wiki projects. And I think there are things for us to learn from the work that is being done in the context of Wiki projects. I can't comment on specifically what I can say that you're exploring that space. One thing is that Wiki projects are very language specific. So they change from language to language. If you want to try to adapt our models and make them better in a given language, we can probably use the information in Wiki projects. I'm not sure whether we can use that for taking information basically across languages or not. But again, definitely something that we are considering to dive in. Awesome, thank you. One or two thoughts from NetROM. I have some thoughts, comments and questions. Two thoughts, consider a Wiki project as a point of entry. And two, for English, French and Russian using ORS parameter injection can allow you to predict quality changes based on a section of k bytes. So let me take the first one. I don't think I understood the second one. So the first one, I guess my question is, can you map Wiki projects to categories? Because if you can do that, then if you know a Wiki project can be represented best by a set of categories, you can basically provide these categories and get from the recommender system a set of sections to be added to articles that belong to these categories. If the answer is no, if the answer is that Wiki projects don't map to specific categories, then I need to learn more and see how we can structure it in ways that we can get more information from Wiki projects. That's the answer to Jonathan NetROM's question. We will see. I've asked them to clarify their second thought. Another question from a Paragos. What approach might they use to find and merge semantically related section titles? Yeah. So Diego, do you want to comment on that topic? On the semantic relatedness of sections. What kind of approach we can use for combining basically sections that have the same meaning? So yeah, we were discussing this, for example, in German, you have two ways to write the same section. I don't remember. I think geography you can write in two ways. And one thing that we were thinking is to look to sections that co-cures with other section names often with the same section name, but doesn't co-cure between themselves. So if you have two things that mean the same, like for example, I don't know, youth and early years, they might co-cure with similar section names, but they will never co-cure between them. So we are thinking to include this to deal with these kind of problems. All right. We have two more questions from NetROM. Question one. Maybe I missed this, but what are the reasons behind using section recommendation as the starting point for recommending to a user that they should add content to an article as compared to other alternatives? Yeah. So I think there were a few reasons. One is that we wanted to choose something that was challenging enough. So starting, for example, very early on with let's say info boxes that has been done quite a bit of work on info box recommendations. And it's not very clear what would exactly be our research contribution except that basically replacing some of the methods that already exist. So in general, we stayed out of info boxes for that reason, at least initially. It has to be challenging. References was something that we considered, but that was complicated because recommending citations in general is a complicated task. If you want to go across projects, this will become even more complicated. And we felt that's too big of a step for the first thing we want to do in this space. We also looked at basically community initiatives and we noticed that the community initiatives and the conversations with the community. And what we learned is that they are actually trying to manually create these templates that create these sections. So one of the biggest motivations was can we do something that basically reduce all these manual work that is going into creating these sections for the community, I think. The collection of all these reasons resulted in us starting from sections. Cool. So we have one more question from NetROM and I think then Daria was in the queue after that. Onboarding newcomers was discussed. What's the idea behind using this to onboard newcomers? Yeah, so the idea is not very well developed. So there is a general idea that is we want to be able to help newcomers. And I think this is a conversation that obviously will need to happen with each community. How do they onboard? What does it mean for them to onboard newcomers? So what are the set of tasks that they're considering? What are the set of guidelines that they want the newcomer to be aware of? I think we are not getting to that level of detail right now. What we're doing is that we're focusing on one aspect of onboarding, which is helping the newcomer learn how to expand an already existing article on Wikipedia. And the question is, can we help the newcomer do this more easily? But also, can we help the people behind these editathons or people who bring people inside the movement to do less of manual work? Basically, again, in the context of expanding articles. I think this is the extent that we are thinking about right now. I think there's nothing really beyond article expansion as far as I remember. And then basically how sections is not enough, right? As I mentioned, it's not enough to tell me, if I go to English Wikipedia and you tell me biographies need to have early life, I have no idea whether I should write three sentences or 20 sentences or 100 sentences. And actually, how should I write an early life section for a biography? So are there ways that we can automatically extract information and provide information to the newcomer that can help them learn some of these very basic components of editing an encyclopedic article in language X? And if we can provide that, let's provide it and see what comes out of it. Awesome. Daria, I think you were next. Sure. I'm actually going to refocus briefly on this issue. And it strikes me that you mentioned that there's a big opportunity for product development around this. And obviously figuring out what are the missing sections and types of content that are missing is a big part of a problem. The other problem is around UX, figuring out what is the right way of serving them in a way that makes sense to the user. And it's an entire area that we haven't even started exploring, right? So my question is about the universe of knowledge that we consider as a source for these recommendations. Obviously, this project is currently looking at everything that exists within the context of the Wikipedia universe, combining that also with Wikipedia. It strikes me that there's a lot of candidate encyclopedic knowledge about topics that already exist in various language editions of Wikipedia that is currently possible to find within the union of all languages that we have, right? And I think an example is the fact that if you look at the work that Wikieta is doing, again, for entities that do exist in at least some language editions of Wikipedia by matching these entities with external knowledge bases, we're basically building the possibilities for identifying types of content, types of knowledge that potentially should be in these languages, but are not there yet. So I was wondering if you had thought with this team about any additional direction for this project looking beyond the universal knowledge that's within Wikipedia's. Yeah, that's a very good question. And it's something definitely that we are aware of. So we are basically kind of amplifying at best whatever comes to Wikipedia and the sources of, for example, if there is bias on Wikipedia and this bias never gets resolved in at least one language, we will never learn about it, right? So how do you come out of this system and try to inject more content from outside? I think one of the things that we are kind of very carefully keeping an eye on is Wikidata. So for us, if a change happens in Wikidata, ideally we would have a system that would be able to basically map that change to corresponding sections of the article and can provide those as recommendations. So you can think of Wikidata is kind of for us is the entry point to more types of knowledge that can be added. And I think it's almost like a separate problem to think about how to populate Wikidata kind of in ways that are sustainable with the kind of content that we want to have across projects. It's an important question, but I think it's potentially outside of the scope of this project at this moment. But I'm really interested in generally exploring that problem, like how can we think about Wikidata? How can we expand some of the capacities that are happening right now there to help reduce bias and content gaps by connecting Wikidata to more external sources? I think once the information is in Wikidata, it becomes easier for systems like this to basically speak with Wikidata and get that content and model it and figure out where in an article it should go to. This kind of system, I think speaking with outside of Wikidata in practice will be really hard because basically if you look at, for example, data sources, they are maintained in different ways. It's a highly distributed system outside already in the Wikimedia world, but when you go outside, like how data is maintained, how data is surfaced. I'm not sure how reasonable it is to expect that a recommender system in Wikimedia would talk with those systems directly. I would expect that to happen through Wikidata. Sure. Just to give one example, just to clear up on this. The example in mind is basically the identifier mapping that exists in Wikidata, which basically says this topic that exists here exists also in external source or external catalog. We can make a reasonable assumption that the structure of the contents in the external catalog will be consistent across all the entities that are described by these identifiers and maybe one possible direction will be to look at, okay, if we link how to say an encyclopedia, and the encyclopedia speaks about a painter, does that entry by the painter have the same sections and the same type of content that we have in the corresponding Wikipedia-language editions, right? So I think that's going to be an interesting content to explore. Yeah, thanks very much. We have a clarification from NetROM on one of the earlier points about ORROR's parameter injections. They say you can ask ORRORs what values it used for predicting the quality of a given revision and then alter those so that it reflects that a section has been added and it will give you an updated prediction. In other words, you can use it to tell you what its prediction will be if the article was altered in a given way. Since you're suggesting adding sections, this means you could use ORRORs to figure out how the predicted quality change of a user following those suggestions. Yeah, that's a good point, yes, I understand that. No other questions on IRC? I've got a question if we still have a few minutes. Yes. So I didn't hear you address this directly in your presentation, Laila, but I may have been distracted for a minute, so apologies if I'm making you repeat yourself. You talked a bit about bias, but one area of bias that I'm curious about are some of the systemic biases that relate to the way certain topics get written about on Wikipedia. An example for which I've heard a lot of anecdotes about this are that biographical articles about women tend to focus on different aspects of a notable woman's life than biographical articles about an equivalently notable male. So you might have an article about a woman's scientist that had a lot more sections and content related to her personal life or her family life than you might have about an equivalent male scientist. In a scenario like that, are there ways in which we... Are you concerned about the possibility of reifying some of these existing biases or are there particular ways you might think of checking for these biases or correcting for them? Yeah, that's a good question. I think all of these systems will need to have feedback mechanisms. So two things, feedback mechanism and we shouldn't try to become too smart. So I think the strength for a project such as Wikipedia is basically the human, the editor. The question is how deep do we want to go to help this editor to do better? And that's how deep is not clear to me. Giving section recommendations is probably going to pass my bar up. It's probably okay. But how deep you go within the section, for example, are you going to say what kind of content should be added to that section? How deep are you going to go in that front? I think the deeper we go, we basically will have our hands dirtier, deeper, with machines. And the question is how are you going to come out of it? One way would be feedback. User feedback is pretty strong, especially in Wikimedia projects where you have a lot of people who are thinking very critically and are willing to express their opinion and tell us what they think. And I think these systems will need to have feedback mechanisms or otherwise they will fall in loops. Some of it I think is inevitable. But again, if the system is at least designed in ways that can receive feedback from the user and can express that as part of its reasoning when it's giving the next set of recommendations, I think that would be really strong. And the other thing I think is just basically the interpretability. That's a topic that, one reason that we're very excited about these kind of category-based recommendations is that we can clearly say why we are making this recommendation. We can say this article belongs to this category. And this category, articles in this category usually have this section. So at least the editor knows why they're receiving such a recommendation and they can do some reasoning. And I think that's really important to keep in mind as we're moving forward. Awesome, thank you. That's all we've got so far. All right, so I think we're going to call it a day here. Thanks everybody for joining. As a reminder, our next showcase will be in January. We already have a conference speaker, Ian Chen from University of Michigan will be talking about expert engagement experiments in Wikipedia on the 17th of January. We're going to have another speaker by then still to be confirmed. So with that, wish you a happy holiday and see you all in.