10.6092/issn.2531-6133/12238
Technology is disrupting the practice of law and revolutionizing how lawyers work.  This revolution is made more powerful because it is increasingly coupled with a rigorous and scientific approach to the law.  In some ways, law is looking more like a Silicon Valley startup and less like the oak-paneled law firms of the last 200 years.  As law, technology, and science merge, the implications for the profession are wide-sweeping.  This article explores persuasive legal writing, offering new thoughts on what the future will hold.  Specifically, this article pilots a method for applying technology and science to measure, analyze and improve persuasive legal writing, offering it as a proof of concept that anchors the article’s broader, and perhaps more controversial assertion.  Namely, more powerful and refined persuasive legal writing software tools, fueled by artificial intelligence, should and will disrupt and reshape significant portions of the legal space, including how legal writing is taught and how it is produced. The effect will be to view legal writing as more science, and less art. The next set of luminaries won’t rely on anecdote or intuition to teach or create legal writing; they will rely on software and data. 
application/pdf
2531-6133
12238
2
en
https://creativecommons.org/licenses/by/4.0
University of Bologna Law Review
Empirical
Ex Machina: Technological Disruption and the Future of Artificial Intelligence in Legal Writing
Text.Serial.Journal
https://bolognalawreview.unibo.it/
5
Articles & Essays
2021-01-17
2021-01-15
2020-09-19
2021-01-17
294-326
UBLR
John Campbell
2020
10.6092/issn.2531-6133/12238
294
University of Denver Sturm College of Law
2531-6133
2
Empirical
en
326
Ex Machina: Technological Disruption and the Future of Artificial Intelligence in Legal Writing
https://bolognalawreview.unibo.it/article/download/12238/12237
5
    llVOLUME 5                                       llISSUE 2                          ARTICLES & ESSAYS



                                 UNIVERSITY OF BOLOGNA LAW REVIEW
                                 https://doi.org/10.6092/issn.2531‐6133/12238
                                                        A

          Ex Machina: Technological Disruption and the Future of
             Artificial Intelligence in Persuasive Legal Writing
                                                                      †
                                                JOHN CAMPBELL


                                                     ABSTRACT

    Technology is disrupting the practice of law and revolutionizing how lawyers work. This
    revolution is made more powerful because it is increasingly coupled with a rigorous and scientific
    approach to the law. In some ways, law is looking more like a Silicon Valley startup and less like
    the oak-paneled law firms of the last 200 years. As law, technology, and science merge, the
    implications for the profession are wide-sweeping. This article explores persuasive legal writing,
    offering new thoughts on what the future will hold. Specifically, this article pilots a method for
    applying technology and science to measure, analyze and improve persuasive legal writing,
    offering it as a proof of concept that anchors the article’s broader, and perhaps more
    controversial assertion. Namely, more powerful and refined persuasive legal writing software
    tools, fueled by artificial intelligence, should and will disrupt and reshape significant portions of
    the legal space, including how legal writing is taught and how it is produced. The effect will be to
    view legal writing as more science, and less art. The next set of luminaries won’t rely on anecdote
    or intuition to teach or create legal writing; they will rely on software and data.


                                                     KEYWORDS
                             Legal Technology; Legal Research; Legal Writing; Empirical




†
     John Campbell is the co-director of the Denver Empirical Justice Institute at the University of Denver Sturm
    College of Law (US), where he is also a member of the Lawyering Process faculty. He gives particular thanks to
    Dr. Catherine Durso of the University of Denver for her work on the statistics presented in this paper. Without
    her, this article would not be possible. He also thinks Kensye Wood and Megan Largson, his research assistants,
    for their dedicated work.



                                                         294
2020]                              UNIVERSITY OF BOLOGNA LAW REVIEW                                                                [Vol. 5:2


                                              TABLE OF CONTENTS

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . .         .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   295
1. Technological Revolution in Law . . . . . . . . . . . . . . .           .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   297
        1.1. Examples from Other Legal Settings . . . . . . . .            .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   297
        1.2. Revolution in Legal Research and Writing Too . .              .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   299
2. Demystifying Persuasive Writing through Technology . . .                .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   303
        2.1. What the Legal Writing Experts Say . . . . . . . .            .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   305
        2.2. Early Empirical Research Regarding Legal Writing                  .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   307
        2.3. Focus on Judicial Writing . . . . . . . . . . . . . .         .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   308
           2.3.1. Intensifiers . . . . . . . . . . . . . . . . . . .       .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   309
           2.3.2. Readability . . . . . . . . . . . . . . . . . . .        .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   310
3. The Pilot Empirical Study . . . . . . . . . . . . . . . . . . .         .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   312
        3.1. United States Supreme Court Briefs are Simpler .              .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   316
        3.2. Writing Style Correlates to Winning . . . . . . . .           .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   319
4. Improving the Method and Tools Available . . . . . . . . .              .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   321
5. Implications for the Profession in the Future . . . . . . . .           .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   322
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . .         .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   .   325




        INTRODUCTION

Empirical legal writing studies, powered by new technologies, will fundamentally disrupt
and revolutionize how we think about persuasive legal writing. This article argues,
through a pilot study that serves as a proof of concept, that, over the coming years, the
fog of advice about persuasive legal writing style can largely be cleared by developing
better tools to measure persuasive legal writing and better methods for studying the
effect of legal writing on outcomes. It argues that, as persuasive legal writing becomes
more science and less art, legal writing software powered by artificial intelligence1 will
disrupt a variety of fields, including how legal writers create briefs, the legal insurance
industry, legal finance of cases in litigation, and how legal writing is taught to students.
            We can and should move away from anecdote and assumption, and towards
software and data. For years, professors (including me), legal writing gurus (like Bryan
Garner who has made millions teaching legal writing), and judges (like the late Antonin
Scalia) have talked about “how” to write effectively. But the truth is that this advice is
largely untested – and as some law students would gladly tell you – too often
inconsistent.
1
    A.I. is an umbrella term that covers a range of technologies that learn over time as they are exposed to more
    data. PEDRO NAVA ET AL., ARTIFICIAL INTELLIGENCE: A ROADMAP FOR CALIFORNIA (Little Hoover Commission 2018),
    https://lhc.ca.gov/sites/lhc.ca.gov/files/Reports/245/Report245.pdf. A.I. is the quality of any computer
    system (data, algorithms, analytics, bots, etc.) the ability to sense, reason, adapt, learn, and understand just
    like humans can. Id. In the deeply developed sectors, A.I. technology can encompass the ability to reason
    through to conclusions and learn to adapt specific outputs or behaviors to circumstances. Id.


                                                         295
                                        EX MACHINA


For years, I have told my law students that if you were to put ten world-renowned
appellate lawyers in a room, gave them the same legal problem, and then had them write
briefs, those briefs would be profoundly different. To be fair, some of this variation is a
result of options in the legal analysis sphere. But assume for a moment you gave the
attorneys the same cases, the same strategy, and the same frames. Assume you even gave
them the general order in which the arguments would be presented. The briefs would
diverge nonetheless.      The writing style among the briefs would vary, sometimes
significantly. To fully understand this notion, you only need to read a few briefs for the
United States [hereinafter U.S.] Supreme Court. World-renowned attorneys with proven
track records take markedly different approaches in their written advocacy, so much so
that their styles are a sort of fingerprint and the author can be guessed by the style and
tone alone.
        The same variance occurs if you put ten legal research and writing professors in
a room. Within my own department, we have engaged in rigorous debates about how
students should be taught to write during our monthly meetings. We all think we know
the “truth” about what lessons or writing methods work. The picture is no different if you
read books on legal writing, or if you attend conferences dedicated to the art.
        Why are there such marked disagreements? The pat answer is that legal writing
is complex, and personal preferences dominate. We might even say authors have to be
themselves, and find the approach that works for them. I take a different view. We see
variance in what people treat as “good” legal writing because we suffer from a severe
deprivation of data.      Like ancient people performing rain dances because their
understanding was limited by the availability of knowledge, we argue about what
“works” in persuasive legal writing because we do not actually know. Sure, we have
hunches. And to be fair, some of these are formed over years of experience, making them
more like educated guesses or very crude statistical inferences. But there is little hard
data upon which to draw any salient conclusions. And absent hard data, the best we can
do is guess.
        Law students reading this might be nodding along as they have sensed the
inherent ‘squishiness’ (to use a technical term) of the advice their law professors give.
Similarly, associates forced to write for more than one partner may too smile in
agreement, as they have been forced to write in two styles to please two partners – both
of which are sure they know how a good brief is written. Or maybe even the professors
are quietly agreeing, as they have read books like Garner and Scalia’s co-authored book
on writing, Making Your Case: The Art of Persuading Judges, in which the authors openly
disagree on a variety of stylistic choices.


                                              296
2020]                            UNIVERSITY OF BOLOGNA LAW REVIEW                                      [Vol. 5:2


We can do better. The remainder of this article explores these issues, suggests future work,
and discusses what the future of persuasive legal writing should look like in the twenty-
first century.




      1. TECHNOLOGICAL REVOLUTION IN LAW

No matter where you look in the practice of law, a combination of technology and an
increasing scientific rigor are changing how lawyers work. This change began before
COVID-19, but the pace has accelerated as more lawyers work remotely, cases are
increasingly decided on briefs and not oral argument, and in general, the practice of law
embraces technology at a faster rate. A few examples make the point and help predict
how technology will do the same to persuasive legal writing.



    1.1. EXAMPLES FROM OTHER LEGAL SETTINGS

Deposition Transcripts: Highlighters and underlining depositions was the norm well into
the 2000s. The more advanced lawyers cut depositions using Microsoft Word. Now, a
new generation of lawyers uses Transcript Pad.2 This application allows them to read a
deposition on their iPad, tap the parts they want to highlight, assign them flags with
topical names, and then print a report for any topic – or all topics.3 That report identifies
the page and line, and produces all the text.4 For a practitioner creating one summary
judgment, this could save two dozen hours of lawyer time.5
          Focus Groups: For decades, before a trial, lawyers assembled focus groups. They
presented their case, asked questions, and had the mock jurors deliberate. Today, massive
online samples of on-demand workers are replacing these methods.6 With big samples,
precise measures of case value, A/B testing of trial strategy, and jury analytics identifying
2
  Virginia H. McMichael, Using the Latest Technology to Tame the Appellate Record and Produce Better Briefs, 41 PA.
  LAW. 48 (2019).
3
  Robert Ambrogi, 42 Essential Apps for Trial Lawyers in 2016, NAT’L L. REV. (Mar. 16, 2016), https://www.natlawr
  eview.com/article/42-essential-apps-trial-lawyers-2016. See also Stephan Futeral, From Toys to Tools Essential
  Tablet Apps for Lawyers, 49 Tenn. Bar J. 14, 15 (2013) (mentioning the many apps available to lawyers to work
  on their iPad).
4
  Ambrogi, supra note 3 (the app “enables you to store, organize, review and annotate all your transcripts on
  your iPad…[fl]ag and highlight important sections and assign issue codes”).
5
  Court requires parties to submit the page and lines for all cited depositions, but most courts prohibit
  including the entire deposition. In the past, cutting the depositions has consumers massive amounts of
  lawyer or paralegal time.
6
  Carol L. Bauss, Speech at the American Association for Justice 2018 National Convention: Technology Use in
  Focus Groups and Jury Selection (Jul. 27, 2018) mentioning two tech tools: (1) software, “Voltaire,” which


                                                       297
                                                  EX MACHINA


ideal jurors is possible.7 Emerging companies in this work, which has emerged in only
the last three to five years, are already reporting involvement in over one-half billion in
verdicts.8
           Trial Presentation: For trial, lawyers often hired a videographer and a technologist
for trial.9 The videographer cut all the videos. The technologist set up the screen, ran
the projector, organized documents to be displayed to the jury, and more. Meanwhile,
the paralegals managed boxes of documents to be shown to witnesses, with copies for
opposing counsel and another for the court.                        Fast forward to today, where a solo
practitioner can download software on their laptop or an app on their iPad that will
manage documents, allow for calling out documents on the screen, highlight text for
witnesses, allow the attorney to label all exhibits, and to distribute them to the court and
opposing counsel.10
           Intellectual Property: Within the field of intellectual property, lawyers who evaluate
patents now regularly rely on a variety of software to do their work.11 They use programs
that analyze the patent application language to determine how it compares to past granted
and denied claims.12 They analyze the grant rate by department and examiner, and they
manipulate patent language to optimize the likelihood of success.13



   performs searches in databases and on the web of prospective jurors, and (2) online focus groups or surveys.
   “Both technologies save time and money and provide valuable insights into how jurors will view a case.” Id.
 7
   Ann T. Greeley, New Online Methods for Jury Research, ABA PRAC. POINTS (Jul. 31, 2018), https://www.american
   bar.org/groups/litigation/committees/products-liability/practice/2018/new-online-methods-for-jury-res
   earch/. See also Murray Ogborn & Theresa Zagnoli, Future trends and potential of focus groups, in 3 Litigating
   Tort Cases (2019).
 8
   See e.g. www.empiricaljury.com.
 9
   See Frank L. Branson, Types of demonstrative evidence—Video, in 4 Litigating Tort Cases (2019) (discussing the
   potential needs for a videographer);Philip Beatty, The Genesis of the Information Technologist-Attorney in the Era
   of Electronic Discovery, 13 J. TECH. L. & POL’Y 261, 262 (2008) (an older discussion on the use of technologists in
   a time where eDiscovery was just starting to ramp up).
10
   TrialDirector 6 for laptop and TrialDirector for iPad are both widely popular programs. L. David Russel
   & Jeffery A. Atteberry, Pros and Cons of Trial Presentation Software Programs, LAW 360 (Apr. 7, 2014),
   https://jenner.com/system/assets/publications/12930/original/Russell_Atteberry_Law360_April_2014.pd
   f?1397055443. With TrialDirector 6, attorneys “can call up exhibits quickly and easily with the use of ‘hot
   key’ shortcuts”; they can “also ‘call out’ and highlight selected text, and make numerous other annotations
   on the fly.” Id. The program handles documents well and can also be used for showing and editing deposition
   video. Id.
11
   Victoria Hudgins, Eyeing Patent Market, Casetext Moves to Expand Its CARA Research Platform, LAW.COM (Oct.
   23, 2019), https://www.law.com/legaltechnews/2019/10/23/eyeing-patent-market-casetext-moves-to-
   expand-its-cara-research-platform/?slreturn=20200113163409(CARA Patent uses A.I. technology developed
   by Casetext to instantaneously reveal the most applicable cases and other IP guidelines).
12
   Id. Intellectual property A.I. programming takes the citations and key terms found within the patent and
   uses patent-specific motion filters, co-reference evaluation, and patterns in PTAB opinions to discover the
   most pertinent results.
13
   Id. See also Edgar Rayo, A.I. in Law and Legal Practice – A Comprehensive View of 35 Current Applications, EMERJ
   (Nov. 21, 2019), https://emerj.com/ai-sector-overviews/ai-in-law-legal-practice-current-applications/
   (discussing other A.I. in IP practice, like Lex Machina to analyze opponent’s arguments).


                                                         298
2020]                           UNIVERSITY OF BOLOGNA LAW REVIEW                                    [Vol. 5:2


Practice Management: The managing of a legal practice is a multifaceted and sometimes
overwhelming aspect of any lawyer’s job. The basic features needed to run a successful
practice at any size include document and task management, time tracking, bookkeeping
and billing, and keeping communications secure.14 Most commonly, firms and solo
practitioners have used separate software programs for each of these categories.
However, practice management software is evolving to address all legal management
needs in one platform.15 Lawyers who have streamlined document assembly, automated
workflow, and have access to useful reports increase their productivity significantly.16
          From these examples, we see that a combination of technology and data analysis
are driving change. The goals are simple: do better work in less time. Lawyers demand
these products, and companies are increasingly happy to invest in making them, as they
see an emerging, lucrative market.



     1.2. REVOLUTION IN LEGAL RESEARCH AND WRITING TOO

Although it may not be obvious now, technology will similarly revolutionize legal
research and writing.          The influence is particularly strong in legal research.                    The
revolution began before COVID-19, but the pandemic has certainly accelerated the pace.
Some attorneys were early adopters who viewed technology as a way to save time and
improve results. However, lawyers and the legal field are notoriously resistant to change.
COVID-19 is changing that. Lawyers are forced to take depositions by video, stodgy
judges are holding hearings by Zoom, and in general, attorneys are recognizing that
technology is not a luxury – it is a necessity.
          Regarding legal research, the first steps towards real change are happening now.
A host of new tools are emerging to research legal questions.17 Instead of searching for
keywords and using Boolean terms to build the search logic, new programs work in




14
   Nicole Black, The Ins and Outs of Law Practice Management Software, ABA J. (2019),
   https://www.abajournal.com/news/article/the-ins-and-outs-of-law-practice-management-software.
15
   See e.g., Clio, https://www.clio.com/. Clio has emerged as one of the most comprehensive practice
   management platforms available. Along with a robust list of typical management features, it offers client
   relationship management (client intake, appointment booking, email automation, etc.) and integrates
   seamlessly with other popular applications like Google Apps, email, and Dropbox.
16
   See Tom Caffrey, Law Practice Management Systems, 35 GPSolo 61, 62 (2018) (discussing technology options and
   principles of good practice management); Joe Forward, Prioritize Efficiency, Maximize Time: The Economics of
   Law Practice, 91 Wis. Law. 26, 33 (2018) (discussing a survey on how law offices have used legal management
   service providers to address client demands and time-use patterns).
17
   To name a few: Bloomberg Law, Casemaker, Casetext, Fastcase, Findlaw, Justia, LexisNexis, MyCase Inc., Ross
   Intelligence, Westlaw Edge, and so much more.


                                                     299
                                                EX MACHINA


different ways.18 They are rooted in algorithms and natural language recognition.19
These processes pull context from uploaded documents and build connections to core
issues.20 The programs, many of which feature artificial intelligence [hereinafter A.I.],
find patterns in facts, procedural history, and citations, all in a matter of seconds.21
          Tools like Westlaw and Lexis have eschewed traditional searches where users
build their own search terms, instead relying heavily on “natural language” searches
rooted in algorithmic decision-making that functions like Google searches.22 The
algorithms examine language of cases, but they also consider what cases other lawyers
have clicked on when running similar searches, the frequency of citation, and more to
rank results.23
          Using A.I. in legal research cuts the time spent sifting through case law, or the
cases cited by an opponent, to a fraction of what it once was, all while being more efficient
and relevant.24 In this way, software is already improving the legal content of briefs.
          Refined legal research is bleeding into the production of better briefs, by refining
their legal content.          Casetext’s brief-analysis software, called CARA (Case Analysis

18
   An example of an A.I. feature in Westlaw is Folder Analysis, which is a feature that is driven by the
   researcher’s interaction with the materials found that are placed into a research folder. Nicole Black, Legal
   Research and A.I.: Looking Toward the Future, ABOVE THE LAW (July 27, 2017), https:abovethelaw.com/2017/07/le
   gal-research-and-ai-looking-toward-the-future/. After the researcher has designated some documents to a
   specific folder, the contents of the folder are analyzed, and additional cases are recommended based on the
   key issues identified by the analysis.
19
   Id.
20
   Mike Whelan Jr., What’s Left for Lawyers?, ABA TECHREPORT (Dec. 11, 2019), https://www.americanbar.org/gro
   ups/law_practice/publications/techreport/abatechreport2019/casetextsponsored/.
21
   Id.
22
   Michael Mills, Artificial Intelligence in Law: The State of Play 2016, THOMSON REUTERS, (Mar. 24, 2016),
   https://www.neotalogic.com/wp-content/uploads/2016/04/Artificial-Intelligence-in-Law-The-State-of-Pl
   ay-2016.pdf.
23
   Although there are several factors that go into ranking the search results in each database, there is a lot
   of secrecy when it comes to each platform’s algorithm. Different research databases dispense strikingly
   different search results. Many speculate that relevancy, key terms, and other factors play a part in the
   algorithm, but no one truly knows. SeeSusan Nevelow Mart, The Algorithm as a Human Artifact: Implications for
   Legal [Re]Search, 109 LAW LIBR. J. 387 (2017)(comparing search results using the search algorithms in Westlaw,
   Lexis Advance, Fastcase, Google Scholar, Ravel, and Casetext). Mart narrowed in on the issue that legal
   research has been a struggle, requiring redundancy in searching because these different algorithms yield
   vastly different results. Id. at 390. Mart attributes the variation in search results using the same search
   terms to the biases inherent in the algorithms, as humans are the ones who essentially code the algorithms
   and build bias into the systems. Id. at 394.
24
   See Stephanie Wilkins, The Key to Crafting A Winning Argument? Context, Above the Law (Jan. 25, 2019),
   https://abovethelaw.com/2019/01/the-key-to-crafting-a-winning-argument-context-2/?rf=1.
   In 2018, LexisNexis launched Context, which analyzes the language of specific judges’ opinions to
   detect cases and arguments each judge views as persuasive. Bob Ambrogi, ‘Context,’ Launching Today
   from LexisNexis, Applies Unique Analytics to Judges and Expert Witnesses, LawSites (Nov.            29, 2018),
   https://www.lawsitesblog.com/2018/11/context-launching-today-lexisnexis-applies-unique-analytics-jud
   gesexpert-witnesses.html. LexisNexis acquired Ravel Law, which initiated these original analytics; Ravel’s
   tools are incorporated into Context within the Lexis Advance legal research platform. Id. Context originates
   the data from court documents and uses the data to predict how likely an argument is to prevail, how judges
   will rule on expert testimony, and to output language federal judges use most often to decide motions. Id.


                                                      300
2020]                             UNIVERSITY OF BOLOGNA LAW REVIEW                                      [Vol. 5:2


Research Assistant), is a good example. Upon uploading any legal document into the
program, latent semantic analytics25 pull case law from a multitude of databases and
produce results directly related to issues implicated by the document.                         26   Importantly,
this includes cases that have been overlooked or not cited in the text.27                                 This is
particularly helpful in drafting a response to briefs submitted by opposing counsel, as it
quickly identifies cases they missed or intentionally omitted.28
           Clerk from Judicata is another A.I.-powered brief analyzer, with a focus on
evaluating the strengths and weaknesses of arguments in an uploaded document.29
According to the advertisements, it is designed to “increase the chances of winning
motions,” Clerk assesses the arguments, drafting, and context of a brief compared to
cases that have historically proven to be more favorable for one side or another.30 The
program aims to help lawyers craft briefs that present logically favorable cases, along
with arguments that have a strong history of being followed.31
           While CARA was the first A.I.-powered search feature to come onto the scene,
many more have followed. Other brief analyzers include:
     a. EVA from ROSS Intelligence—scans the brief to check the authority cited to
        determine whether all citations are still good law;32
     b. Vincent from vLex—analyzes briefs in both English and Spanish, and is often used
        as a foreign-law resource;33
     c. Quick Check from Thomson Reuters—advertises delivering a limited set of the most
        highly relevant results from uploaded briefs;34 and
     d. Brief Analyzer by Bloomberg.35
25
   See Shannon Brown, Peeking Inside the Black Box: A Preliminary Survey of Technology Assisted Review (TAR) and
   Predictive Coding Algorithms for eDiscovery, 21 SUFFOLK J. TRIAL & APP. ADVOC 221 (2016)(latent semantic analysis
   is a natural language processing technique that analyzes relationships between a set of documents and the
   terms they contain by producing a set of concepts related to the documents and terms).
26
   Pablo Arredondo & Chelsea Strauss, Putting Casetext’s CARA to the Test, Stan. L. Sch. Blogs (Dec. 9, 2016),
   https://law.stanford.edu/2016/12/09/putting-casetexts-cara-to-the-test/.
27
   Id.
28
   Id.
29
   See e.g. Judicata, https://www.judicata.com/demo/clerk/report (last visited Feb. 8, 2020).
30
   Beth Hoover, Introducing Clerk, Judicata (Oct. 5, 2017), https://blog.judicata.com/introducing-clerk-
   848abbed8fd3.
31
   Id.
32
   See Ross Intelligence, https://rossintelligence.com/ (last visited Feb. 8, 2020). Not only does the system
   check all the authority for good law, but then the program generates its own legal research to find better
   cases to support the overall position.
33
   Bob Ambrogi, Vincent Joins CARA, EVA and Clerk as the Latest A.I.-Driven Research Assistant, LawSites (Sept.
   20, 2018), https://www.lawsitesblog.com/2018/09/vincent-joins-cara-eva-clerk-latest-ai-driven-research-
   assistant.html.
34
   Bob Ambrogi, A.I.-Driven Brief Analysis Comes to Westlaw, But Does It Differ from Competitors?, LawSites (July
   12, 2019), https://www.lawsitesblog.com/2019/07/ai-driven-brief-analysis-comes-to-westlaw-but-does-it-
   differ-fromcompetitors.html.
35
   See Bloomberg, https://pro.bloomberglaw.com/brief-analyzer/ (last visited Feb. 7, 2020).


                                                        301
                                                   EX MACHINA


Another area of continued innovation revolves around textual analysis software that
focuses not on case law, but on identifying patterns with documents (or sets of
documents).
           The use of A.I. is most common in e-discovery.36 Over the last few decades or
more, the use of A.I. to sift through documents in discovery has grown exponentially. In
that setting, A.I. lets lawyers work through tens of thousands of documents quickly by
automatically searching for common language, buzz words, repetition, confidential
information, and more.37
           As e-discovery emerged, lawyers grappled with growing data volumes and the
time-consuming job of reviewing that data. Technology assisted review [hereinafter
T.A.R.] allows lawyers to review a “seed-set” of a large collection of documents after
which the system will automatically go through the complete collection.38 Not only does
this save teams of lawyers’ countless hours, but often T.A.R. systems return more
accurate and complete results than a team of humans would.39 E-discovery incorporates
predictive coding and A.I. tools, like natural language capabilities and machine
learning.40 These tech improvements in review processing make reviewing more types
of data possible, even those data sets that are unstructured.41 The development of these
complex textual analysis tools is essential for document review, but it is also innovation
that has quickly carried over into new fields.
           For example, a growing sector in “Legal Tech” deploys A.I. to examine contracts
and to draft them.42             At their core, these programs compare existing contracts to
thousands of past contracts to identify similarities and differences, and they suggest core
terms for new contracts.43 But beyond that, they also “learn” from the contracts they
review, so that the advice they offer evolves as contracts evolve.44


36
   Sergio D. Becerra, The Rise of Artificial Intelligence in the Legal Field: Where We Are and Where We Are Going, 11 J.
   BUS. ENTREPRENEURSHIP & L. 27, 39 (2018).
37
   Sharon D. Nelson & John W. Simek, Running with the Machines Artificial Intelligence in the Practice of Law, OR. ST.
   B. BULL., Dec 2017, p 22, 23-24.
38
   Kent B. Goss et al., Welcome to Your New War Room, 34 Westlaw J. Corp. Officers Dir. Liab. (2019).
39
   Id.
40
   Jamie J. Baker, 2018 A Legal Research Odyssey: Artificial Intelligence as Disruptor, 110 LAW LIBR. J. 5 (2017).
41
   Goss, supra note 38.
42
   Blake A. Klinkner, Artificial Intelligence and the Future of the Legal Profession, WYO. LAW., Dec 2018 p. 26, 28
   (“Natural language processing and machine learning have allowed programs to be developed which may
   analyze large datasets of contractual documents and attendant datapoints, and then learn which contractual
   terms and conditions are best under certain conditions.”). This inherently means that computers are
   drafting the entirety of contracts using data from clients, with the capability to flag potentially negative
   language in an opposing contract for a lawyer to review. Id.
43
   Nicole Black, Here’s the Lowdown on Contract Analytics Software, ABA JOURNAL, (Mar.                        23, 2018),
   http://www.abajournal.com/news/article/heres_the_lowdown_on_contract_analytics_software.
44
   Id.


                                                          302
2020]                             UNIVERSITY OF BOLOGNA LAW REVIEW                                       [Vol. 5:2


Document generation has entered the market with software programs that generate
wills, real estate documents, incorporation documents, and promissory notes.45 Take for
example JPMorgan Chase, who introduced COIN, a contract intelligence system that
reviews commercial loan agreements.46 COIN reduces mistakes and cuts review time
significantly. A due-diligence program created for mergers and acquisitions, DLA Piper,
incorporates the same system.47 Other programs like Kira and Lawgeex have the ability
to suggest edits to contracts based on pre-defined parameters.48
           But these are just early steps, steps that will look humble, even quaint in a few
years – like looking at a Commodore 64 next to an iPhone. Technology has changed the
legal market, and as technology progresses and expands, those changes will reach every
corner of practice. This includes, as this article examines, the tone, style, and overall feel of
briefs. It is beyond the scope of this article to discuss whether this disruption will replace
lawyers. Those topics have been discussed at length by others.49 But it is my opinion that
as legal technology reforms our understanding of persuasive legal writing, we will all be
better for it.




      2. DEMYSTIFYING PERSUASIVE WRITING THROUGH TECHNOLOGY

Improving the way we think about persuasive legal writing is necessary if we are to move
beyond anecdote and contradiction. Conventional wisdom about good legal writing
abounds. Unfortunately, such wisdom is often untested and contradictory. For instance,
as mentioned above, when Bryan Garner and former Justice Antonin Scalia co-wrote a
book, on legal writing, the two brilliant writers could not agree on using contractions,

45
   Stephanie Wilkins, Top 4 Documents Automation Software Tools for 2020 - Reviewed, ABOVE THE LAW (Feb.
   5, 2020), https://abovethelaw.com/?sponsored_content= top-4-document-automation-software-tools-for-
   2020-reviewed (reviewing four document generating tools—Documate, Formstack Documents, PandaDoc,
   and HotDocs—all of which were variations on automating contracts to streamline workflows and increase
   efficiency).
46
   Nelson & Simek, supra note 37.
47
   Nelson & Simek, supra note 37 (“DLA Piper is using artificial intelligence software for due-diligence document
   review in mergers and acquisitions. The software searches text in contracts and then creates a summary and
   an analysis.”).
48
   Lisa Angelo et al., Examples of Artificial Intelligence Systems in Legal, TXCLE-ADVANCED Family L. 30-V (2018).
49
   Dana Remus & Frank Levy, Can Robots Be Lawyers? Computers, Lawyers, and the Practice of the Law, 30 GEO. J. LEGAL
   ETHICS 501 (2017), SSRN, 1, 1- 2 (2015) (discussing the various examples of articles and literature that have
   been written on this robots taking over the role of lawyers). Another those in the camp concerned over
   the rise A.I. in legal practice warn that biases, deception, and malicious actions can occur in applying John
   Levin, Big Data, Artificial Intelligence, and Legal Ethics, CBA REC., 48, Apr.-May 2019.These individuals are that
   humans are still providing the codes and structures that make A.I. function, which can be a breeding ground
   for implicit bias that is even more difficult to uncover.


                                                        303
                                                  EX MACHINA


gender neutral nouns, or whether citations should be relegated to footnotes.50 Similarly,
some lawyers suggest that any use of legalese is unwise,51 while others suggest that
writing too simply makes a brief pedestrian.52 And according to some appealing to
emotion is foolish,53 while opposing authorities emphasize storytelling as a means of
persuasion.54 So, who are we to believe? What styles dominate the upper echelon of legal
writing, and perhaps more important, does writing style matter at all?
           Because writing drives decision making, and as a result is one of the most
valuable skills a lawyer can have, I started answering these questions using empirical
methods to study writing style. The results are preliminary, yet promising. They suggest
that with continued refinement of our methods, writing style and its implication on
decision-making can be measured and understood.
           This conclusion has implications for a variety of industries. For example, it
suggests that companies that insure verdicts may produce refined analytical programs to
better help them contemplate risk. It also suggests, that brief writers could use refined
analytical tools to make their briefs better, potentially increasing chances of winning.
For researchers, it suggests there is a great deal of interesting work we can do to develop
tools and methods for examining persuasive legal writing more effectively. Finally, it
suggests for those of us who teach legal writing that: (a) we can and should refine our
teachings through the lens of hard data and; (b) that soon (or even now) we may provide
tools to our students that will empower them to write better by leveraging technology.55

50
   ANTONIN SCALIA & BRYAN A. GARNER, MAKING YOUR CASE: THE ART OF PERSUADING JUDGES (2008).
51
   See, e.g., NANCY L. SCHULTZ & LOUIS J. SIRICO JR., LEGAL WRITING & OTHER LAWYERING SKILLS 90-91, 93-94 (5th ed.
   2010).
52
   See MARK ADLER, THE OXFORD HANDBOOK OF LANGUAGE AND LAW 67 (Lawrence M. Solan & Peter M. Tiersma
   eds., 2012) (“lawyers have historically believed that traditional legalese is more precise and “plain language
   represents irresponsible over-simplification”). One study in the English education field suggests that such
   skepticism may be warranted. Rosemary L. Hake & Joseph M. Williams, Style and Its Consequences: Do as I
   Do, Not as I Say, 43 COLL. ENG. 433 (1981). The experiment detailed that a group of college English teachers
   gave higher grades to papers with syntactically complex writing than to papers written simply. Id. The
   researchers inferred that the writers of simpler prose may have been perceived as naive and less intellectual
   than the writers of the complex prose. Id. at 50-51.
53
   Todd E. Pettys, The Emotional Juror, 76 FORDHAM L. REV. 1609 (2007) (discussing a large number in the legal
   profession have come to believe “emotions undercut rational decision making”).
54
   See Kenneth D. Chestek, Judging by the Numbers: An Empirical Study of the Power of Story, 7 J. ASS’N LEGAL WRITING
   DIR. 1, 19–22 (2010), 7 J. Ass’n Legal Writing Directors 1, 19–22 (2010) (presenting empirical evidence that, as
   judges and lawyers progress in their careers and gain experience, they increasingly value the narrative in
   the case as a matter of persuasion). On appeal, it is not enough to simply craft a great legal argument. As
   Ninth Circuit Judge Alex Kozinski glibly notes, “[t]here is a quaint notion out there that facts don’t matter
   on appeal-that’s where you argue about the law; facts are for sissies and trial courts. The truth is much
   different. The law doesn’t matter a bit, except as it applies to a particular set of facts.” Alex Kozinski, The
   Wrong Stuff, 1992 B.Y.U. L. REV. 325, 330. In other words, an appellate brief must tell a good story.
55
   This can give instructors guidance when expectations to teach successful writing are compounded with
   first having to teach basic writing skills. Sarah Valentine, Legal Research as a Fundamental Skill: A Lifeboat for
   Students and Law Schools, 39 U. BALTIMORE L. REV. 173, 209-10 (2010). Firms are beginning to conform practice
   sectors around technology and adopt innovative procedures, but technology does not play an integral role in


                                                        304
2020]                           UNIVERSITY OF BOLOGNA LAW REVIEW                                    [Vol. 5:2


Before turning to the study and its implications, it is helpful to first understand some
existing literature, both anecdotal and empirical. The following section provides a look at
“conventional wisdom” for legal writing. Thereafter, an examination of the various ways
empirical researchers are studying writing by lawyers and by judges is undertaken. In
particular, that section addresses studies that consider whether writing style, measured
in a variety of ways, impacts outcomes.



    2.1. WHAT THE LEGAL WRITING EXPERTS SAY

Because of the prevalence of unappealing legal writing and the importance of writing to
lawyers’ work, many legal writing experts give advice for stylish legal writing. Bryan A.
Garner has written numerous legal-style books and articles.56 Other classic legal-style
texts include Plain English for Lawyers57 and Thinking Like a Writer.58
          The extensive literature discussing legal writing lacks a systematic analysis of
the fundamental qualities of good legal writing. However, there seems to be a consensus
among the legal writing experts that the chief qualities of good legal writing are clarity
and conciseness.59
          Legal writing experts emphasize clarity and the avoidance of legalese.60 Justice
Benjamin Cardozo explains, “there can be little doubt that in matters of literary style the
sovereign virtue for the judge is clearness.”61 To echo this point, in their book Making
Your Case, Justice Antonin Scalia and Bryan Garner claim that “one feature of a good style
trumps all others. Literary elegance, erudition, sophistication of expression—these and
all other qualities must be sacrificed if they detract from clarity.”62 As Garner further
explains, “A lawyer should keep in mind that the purpose of communication is to
communicate, and this can not be done if the reader or listener does not understand the

   most lawyers’ education, which is a hurdle for the firm and the law student pining for a job. Survey Report,
   WOLTERS KLUWER, THE FUTURE READY LAWYER: THE GLOBAL FUTURE OF LAW (2019).
56
   See e.g., BRYAN A. GARNER, LEGAL WRITING IN PLAIN ENGLISH: A TEXT WITH EXERCISES (2001). Garner describes an
   aspect of “poor legal writing” is “mak[ing] law students pore over ream upon ream of tedious, hyperformal,
   creaky prose” and fostering “them to pomposity.” Id. at xvii-xviii. Lawyers “learn [their] trade by studying
   reams of linguistic dreck—jargon-filled, pretentious, flatulent legal tomes that seem designed to dim any
   flair for language.” Id.
57
   RICHARD C. WYDICK, PLAIN ENGLISH FOR LAWYERS 3 (5th ed. 2005).
58
   STEPHEN V. ARMSTRONG & TIMOTHY P. TERRELL, THINKING LIKE A WRITER: A LAWYER’S GUIDE TO EFFECTIVE WRITING
   AND EDITING (3rd ed. 2008).
59
   See generally id.; Garner, supra note 56; TOM GOLDSTEIN & JETHRO K. LIEBERMAN, THE LAWYER’S GUIDE TO WRITING
   WELL (1989); Wydick, supra note 57, at 58-60.
60
   GARNER, supra note 56; WYDICK, supra note 57, at 58-60; see generally ARMSTRONG & TERREL, supra note 58; TOM
   GOLDSTEIN & JETHRO K. LIEBERMAN, supra note 59.
61
   BENJAMIN N. CARDOZO, LAW AND LITERATURE (1931), reprinted in LAW AND LITERATURE AND OTHER ESSAYS AND
   ADDRESSES 7 (1986).
62
   SCALIA & GARNER, supra note 50, at 107.


                                                     305
                                                 EX MACHINA


words used.”63 Experts agree, mostly, that writing style full of intricate constructions
and throat-clearing verbiage does not effectively persuade or connect the reader.
           As many in the legal profession know, Bryan Garner is conceivably the leading
legal writing expert, advocating for plain style in writing—clean, coherent, controlled,
and commanding. Garner has likewise recommended that lawyers communicate their
arguments in writing “honestly, clearly, unpretentiously,” using a “natural voice.”64 He
advises that lawyers use a “literate, precise, but relaxed style.”65
           Garner endorses the following as a “good test of naturalness: if you wouldn’t say
it, then don’t write it.”66 He advises attorneys to “try reading your prose aloud” during
editing “to see whether you’d actually say it the way you’ve written it.”67                               Garner
resounded this tactic in his book, The Winning Brief, where he offered 100 tips for the
building blocks of brief writing.68
           Garner argues that writing is effective and persuasive when clear, plain arguments
are formulated in a concise manner.69 “[T]he first and last secret of a good style consists
in thinking with the heart as well as with the head.”70 Legal style is important and can
affect the impression the writing leaves on the reader.



63
   BRYAN A. GARNER ET AL., THE REDBOOK: A MANUAL ON LEGAL STYLE 183 (2d ed. 2002).
64
   See Bryan A. Garner, An Approach to Legal Style: Twenty Tips for the Legal Writer, 2 SCRIBES J. LEGAL
   WRITING 1, 1 (1991) [hereinafter Garner, An Approach to Legal Style] (“Use words and phrases that
   you know to be both precise and as widely understood as possible.”); see also Bryan A. Garner,
   The Question of Voice How to Bring a More Conversational Style to Your Writing, ABA. J. (Dec. 1, 2016),
   https://www.abajournal.com/magazine/article/garner_conversational_writing [hereinafter Garner, The
   Question of Voice]:
                The other day a lawyer asked me: “Isn’t one of the hardest things about editing well
                learning to improve the writing while not changing the writer’s voice?” I said no:
                When editing most lawyers’ work, I have little regard for the writer’s voice because
                most lawyers haven’t cultivated a discernible voice. What all legal writers should
                strive for is to be the voice of reason.
65
   Garner, An Approach to Legal Style, supra note 64.
66
   BRYAN A. GARNER, supra note 56.
67
   Id.
68
   BRYAN A. GARNER, THE WINNING BRIEF: 100 TIPS FOR PERSUASIVE BRIEFING IN TRIAL AND APPELLATE COURTS 226
   (3rd ed. 2014) (discussing Tip #28: never write a sentence that you couldn’t easily speak; “Try to get your
   speaking voice in your writing . . . In talking, you tend to use short sentences, plain words, active voice, and
   specific details . . .” (quoting DANIEL MCDONALD & LARRY BURTON, THE LANGUAGE OF ARGUMENT 238 (Houghton
   Mifflin 1986))).
69
   Garner also advises that judges and lawyers should place citations in footnotes to ensure that readers aren’t
   getting lost in clunky citations or unwieldy string citations with long explanatory parentheticals, but he
   is strongly repulsed by the use of substantive footnotes, like this one. Garner, supra note 63, at 176-99
   (elaborating on Tip 24-Put all your citations in footnotes, while saying in the text what authority you are
   relying on. But ban substantive footnotes).But see Richard A. Posner, Against Footnotes, CT. REV., Summer
   2001, at 24, 24 (taking issue with Garner’s suggestion that all legal writers put citations in footnotes when
   drafting briefs or opinions).
70
   Garner, An Approach to Legal Style, supra note 64 (quoting ARTHUR QUILLER-COUCH, ON THE ART OF WRITING 291
   (1916)).


                                                       306
2020]                              UNIVERSITY OF BOLOGNA LAW REVIEW                                         [Vol. 5:2


Although none of these “truths” sound objectionable, it is notable that Garner does not
provide citations for these truths. Indeed, he likely cannot. The research does not exist.
It would be unthinkable in other fields to shape such an important part of that
profession’s work on anecdotes. Structural engineers do not rely on their gut. And we do
not want doctors performing surgery based on what other doctors say is effective. We
expect measures, analysis, and best practices. Why should it be any different for legal
writing?



     2.2. EARLY EMPIRICAL RESEARCH REGARDING LEGAL WRITING

The study of legal writing using empirical methods is growing and is likely to continue.71
With the continued development of software that can gather briefs, analyze them, and
be coded to engage in a variety of analyses, there are amazing new avenues to explore.
Entire articles have been written explaining new methods, exploring those methods, and
providing early, tantalizing results.72 The existing research covers a variety of topics,
including how the use of “intensifiers” impacts outcomes,73 whether readability
(measured by a few common, simple readability scores) predicts outcomes,74 and
whether citations predict results. Other articles have engaged in a more ambitious and
creative analysis. For example, one article treated each precedent by a Supreme Court
justice as their “output” and then measured the influence of that output by how often
those cases are cited. It used that data to measure which justices, and categories of
justices, have the most influence.75 Other papers examine Supreme Court decisions
using simple measures, such as opinion length.76 Specific to the instruction of legal
writing, there is even a textbook that attempts to root instruction in early findings,
rather than shared (but often unexamined) beliefs among legal writing teachers. The
book, The Science Behind the Art of Legal Writing, draws on a variety of studies.77
71
   See Shaun B. Spencer, Using Empirical Methods to Study Legal Writing, 20 J. LEGAL WRITING INST. 141 (2015). In his
   article, Spencer discusses empirical research methods that contribute to the growing field of research. Id.
   at 184. He highlights just how empirical research positively adds to learning and developing legal writing
   skills. Id.
72
   See Chad M. Oldfather, Joseph P. Bockhorst & Brian P. Dimmer, Triangulating Judicial Responsiveness: Automated
   Content Analysis, Judicial Opinions, and the Methodology of Legal Scholarship, 64 FLA. L. REV. 1189, 1238 (2012).
73
   See Lance N. Long & William F. Christensen, Clearly, Using Intensifiers is Very Bad - Or Is It?, 45 IDAHO L. REV. 171
   (2008).
74
   See Lance N. Long & William F. Christensen, Does the Readability of Your Brief Affect Your Chance of Winning on
   Appeal?, 12. J. APPELLATE PRAC. & PROCESS 145 (2011).
75
   See William M. Landes, Lawrence Lessig & Michael E. Solimine, Judicial Influence: A Citation Analysis of Federal
   Courts of Appeals Judges, 27 J. LEGAL STUD. 271 (1998). (measuring the influence of particular judges based on
   the number of times their opinions are cited).
76
   See Ryan C. Black & James F. Spriggs II, An Empirical Analysis of the Length of U.S. Supreme Court Opinions, 45
   HOUSTON L. REV. 621 (2008).
77
   CATHERINE J. CAMERON & LANCE N. LONG, THE SCIENCE BEHIND THE ART OF LEGAL WRITING (2015).


                                                          307
                                                 EX MACHINA


     2.3. FOCUS ON JUDICIAL WRITING

An article on Supreme Court opinion length examined the causes of increased opinion
length, noted that that growth was not linear but exhibits periods of growth and
contraction, and concluded that longer opinions – controlling for many other factors –
are cited more often and carry more precedential force.78
           Another article examined how citation of opinions could measure the influence
which authoring judges had with his or her peers.79 It reached several interesting
conclusions, including that older judges were more influential than younger judges,80
that law professors are among the most influential judges,81 and that whether a judge
had served as a judge in another court did not alter influence.82
           Yet another article provided tantalizing data about how courts use, or ignore
briefs.83 The authors explored whether courts are “responsive” to the briefs they read in
a variety of ways, including measuring how often they cite cases first cited by the
litigants.84     The results were surprising, for example, they found that just roughly
one-third of total citations can be found in the appellant/appellee briefs in the Circuit
they studied,85 and that courts adopt reply briefs the least, raising questions as to why.86




78
   Black & Spriggs, supra note 76, at 634-40 (showing a dramatic increase in the Supreme Court opinion lengths
   in the second half of the 20th century in comparison to historical norms). “While the median length of
   the Court’s majority opinions hovered around 763 words for the first twenty years of its existence, the
   same quantity more than quintupled to 4,250 words for the most recent twenty-year period.” Id. at 634.
   A contributing factor of this increase may exist in Justices feeling compelled to justify their position to offer
   the best guidance in comparison to other opinions and even refute those dissenting opinions. Id. at 629
   (Justice Powell stated that he “prefer[red] ‘lean’ opinions, but [[that] it is important to meet honestly and
   fairly the serious arguments advanced by the losing side or by a dissenting opinion.”).
79
   Landes, supra note 75, at 271-72.
80
   Id. at 279-80.
81
   Id. at 288. Twenty percent of the top twenty judges were former law professors, including the first and third
   judge. Id.
82
   Id. at 318-19.
83
   Oldfather, supra note 72.
84
   Id. at 1195.
85
   Id. at 1238 (“On average, only 35% of the authorities cited in the court‘s opinions were among those cited by
   the parties, and the court cited just over 16% of the authorities referenced in the briefs.”).
86
   Id. at 1195.


                                                        308
2020]                               UNIVERSITY OF BOLOGNA LAW REVIEW                                          [Vol. 5:2


     2.3.1. INTENSIFIERS

Long and Christensen examined whether what they call intensifiers87 (words like clearly,
obviously, certainly) impacted outcomes in appellate cases.88 They noted that many legal
writing instructors and many manual books on legal writing suggest that intensifiers
should not be used. They cited Chief Justice Roberts, who has condemned the use of such
words in Supreme Court briefs.89 The study measured what the authors called the
intensifier rate (number of intensifiers per page).90 The results were interesting, but
muddy. Intensifiers did not correlate to losing more often, although the data hinted at
the result without being statistically significant.91                          But, surprisingly, the authors
identified situations in which increased use of intensifiers correlated with a higher win
rate.92      This occurred when the judicial opinion also exhibits a higher number of
intensifiers.93 The authors are candid about the limitations of these results, concluding
they cannot draw a causal conclusion.94 An apparent and potential explanation of the
data is simply that some cases are actually obvious and clear. The author does not lose
credibility to point to this truth for the court. And if the court agrees, and perhaps even
views the other side’s position as borderline frivolous, it scolds them with intensifiers. In
this way, one would expect to see increased intensifiers in the appellate brief and in the
opinion.
           Regardless of the final takeaway of the study, it is an example of how statistical
analysis can attempt to measure the effectiveness of legal writing, and perhaps a
cautionary tale of just how complicated the results can be.95




87
   Lawyers’ tendency to advocate in writing could lend to them using intensifiers (i.c. the lawyer really, really,
   really, wants to win for her client). See Elizabeth R. Frost, Cutting the Clutter: Spring Cleaning for Writing, 2013 OR.
   STATE BAR BULL. 15. (“Mark Twain, America’s official authority on everything, advised writers to “substitute
   “damn” every time you’re inclined to write “very”; your editor will delete it and the writing will be just as
   it should be.”).
88
   See e.g., Long & Christensen, supra note 73, at 180.
89
   Disapproving the use of the intensifier “clearly” in briefs for the Supreme Court, Chief Justice Roberts
   snarked that, if the case were that clear, it would not be before the Court. Id. at 172.
90
   Id. at 181.
91
   Id. at 173. “The degree of intensifier use by the writer of a legal brief is a function of the writer’s perception
   of the strength of his or her own argument, relative to the opposing side’s argument.” Id. at 186.
92
   Id. at 181-82.
93
   Id. at 184, n. 55 (“It can even be argued that the high rate of intensifiers in judicial opinions, especially where
   the answer is not clear, serves as a model for high intensifier use by practitioners in similar situations.”).
94
   Id. at 172.
95
   At any rate, correlation is not necessarily causation. However, the fact that intensifiers may correlate with
   losing briefs is enough to give a legal writer pause to avoid using language that lacks any real force or power.
   See Frost, supra note 87 (highlighting the unnecessary repetition in using the phrases “utterly convinced”
   and “very urgent”).


                                                           309
                                                   EX MACHINA


     2.3.2. READABILITY

 Long and Christensen took on readability in a subsequent article.                               They used two
 common measures of readability96 – the Flesch Reading Ease Scale,97 and the
 Flesch-Kincaid Grade level scale.98 Both scales function largely by measuring the length
 of words, and the length of sentences.99 Shorter words and shorter sentences produce
 lower scores. Such scales have been roundly criticized as doing a poor job of measuring
 actual readability.100 But the authors note they are not concerned about such criticism
 because they want to simply measure whether shorter sentences and shorter words
 correlate to better outcomes, reasoning that judges might find such sentences easier to
 read.101
            The conclusion that “readability” does not correlate to outcomes would suggest
 appeals are decided on the merits, and that the method of delivery is largely irrelevant.102
 If this is true, it would suggest all lawyers should spend less time on how they write, and
 simply make sure the content is sound. The authors candidly suggest that readability may
 be “sound and fury signifying nothing.”103
            As explained in my results section, my findings are at least arguably at odds with
 this conclusion. Using more refined tools and measures, I conclude that some writing
 styles (which is a form of readability), do correlate with success. This is likely due to the
 difference in measurement tools.                   The “readability” statistics used by Long and

96
    See Long & Christensen, supra note 74, at 145.
97
    The Flesch Reading Ease scale (FRES) measures the readability of a text on a scale from 0-100 with higher
    scores indicating texts that are easier to understand: FRES = 206.835 - 1.015 Total Words Total Sentences -
    84.6 Total Syllables Total Words See, e.g., Rudolf Flesch, A New Readability Yardstick, 32 J. APPLIED PSYCH. 221,
    223-33 (1948).
 98
    The Flesch-Kincaid Grade Level (FKGL) measures the number of years of education typically required to read
    a text FKRA = 0.39 Total Words Total Sentences + 11.8 Total Syllable Total Words - 15.59. Id.
 99
    See Norman O. Stockmeyer, Using Microsoft Word’s Readability Program, MICH. BAR J., Jan. 2009, at 46. Both of
    these readability scales are found within Microsoft Word.
100
    See K.K. DuVivier, Writing Help at Your Fingertips-Readability Scale, COLO. LAW, Mar. 2001, at 39.(“The
    shortcoming of readability scales is that they can only measure the surface characteristics of words. They
    assume that reading is equivalent to understanding.”). Duvivier explains that computers can count the
    number of words between periods with ease, but the real shortfall is a computer cannot distinguish citation
    sentences from grammatical sentences—the scales measure each period as the end of a sentence. Id.
    “Consequently, a citation sentence, such as 42 U.S.C. § 1983 (1994), is read as four short sentences, which
    can inaccurately boost a text’s readability rating.” Id.
101
    But see Long & Christensen, supra note 74, at 154.
102
    Although the study found no significant correlation between the readability and the success of the briefs,
    this conclusion could merely expose lower caseloads in a specific court and a larger number of law clerks to
    support appellate judges compared to the trial judges. Also, their conclusion could have been “impacted the
    logistic regression analysis, as the inferential logic requires variation in the dependent variable to draw a
    valid conclusion.” see William D. Woodworth, The Ethics and Science of the Legal Writing Art: An Interdisciplinary
    Approach, 67 SYRACUSE L. REV. 329, 341 (2017). See also Long & Christensen, supra note 74, at 156 (failing to
    control for other dimensions of narrative writing in the logistic regression).
103
    Long & Christensen, supra note 74, at 161.


                                                         310
 2020]                             UNIVERSITY OF BOLOGNA LAW REVIEW                                       [Vol. 5:2


 Christensen are better termed “how-many-syllables-appear-in-a-sentence” tools, which
 are crude. For example, the word “intelligent” is far more understood and common than
 “apt”. But the former would score as less readable than the second. Similarly, most
 people understand “phony” or “insincere” but might not know the word “glib.” Beyond
 examples of short words, anything but simple or readable, the law complicates things
 further. Some legal words are terms of art. Failure to use them might shorten the
 sentence, but omitting the words would not improve the readability (or credibility) of
 the author.        Finally, the Microsoft Word readability check scores legal citations as
 sentences.104 The citations affect the readability score of the writing with the amount of
 punctuation.105
            StyleWriter provides a more refined tool that measures how “readable” writing
 is in a variety of ways. This includes scoring words based on their generally accepted
 meaning, rather than their length. It also considers the use of active versus passive verbs,
 how often prepositional phrases are used, and the role of jargon.106




104
    Id.
105
    See Shaun B. Spencer & Adam Feldman, Words Count: The Empirical Relationship Between Brief Writing and
    Summary Judgment Success, 22 J. LEGAL WRITING INST. 61, 81 (2018). (“Removing the citations alleviates the
    risk that the awkward form of legal citation would undermine the reliability of the readability measures.
    When we took a small sample of briefs and ran several common readability tests with and without citations,
    the two sets of readability scores varied wildly and produced significantly different rank-ordering of the
    briefs.”) Another factor to think about is the variation in the number or length of citations used. The citation
    lengths vary based on the number of sources cited and the various reporters used for any internal citation.
    See also Black & Spriggs, supra note 76, at 631 n. 36 (“[M]odern opinions are likely to cite several reporters for
    any given internal citation, whereas earlier opinions will have systematically fewer reporters because many
    did not yet exist.”). Black and Spriggs compared the difference in total calculated length of the “original”
    opinion with the length of a “clean” opinion—to get this “clean” version the authors eliminated many of
    the citations within the opinion based on a list of about 150 reporter citation stems that appeared in their
    data. Id. The two authors found that prior to 1940, the “original” opinion “averaged roughly 65 more words
    than the cleaned version.” Id. However, after 1940, the difference between the two versions of the opinion
    increased by a factor of approximately five, differencing about 315 words.
106
    I am not endorsing StyleWriter as the only tool that can do this work. But it proved effective for my needs.
    StyleWriter is one of the oldest writing analysis programs, and is currently in its fourth version. There are
    several alternatives to StyleWriter. For example, ProWritingAid and SlickWrite are similar to StyleWriter
    in that they produce reports which identify a variety of writing errors, i.e., alliteration, cliches, and poor
    word choice. These programs provide a readability score as well. PaperRater also seems to provide some
    similar analysis, but more limited in scale. It is also worth noting that there are other programs available
    that do not providing an analytical report of writing, but instead attempt to correct mistakes in real time.
    Wordrake, for example, is more business-writing based, edits for brevity and clarity, and is popular among
    lawyers. The Hemingway app will color code errors and offer corrections as a person writes.


                                                         311
                                                 EX MACHINA


          3. THE PILOT EMPIRICAL STUDY

 In this initial study, I sought to explore two core questions. First, I wondered whether,
 with refined tools, it was possible to better describe existing writing styles and
 differences among courts and authors. Second, I wanted to learn whether I could
 identify initial metrics that correlate with outcomes. I stress that this study is more
 proof of concept, than conclusive data. However, the results are promising, suggesting
 that with continued innovation we can more meaningfully measure writing styles, and
 learn what styles are the most effective. I explain the methods and results in the
 following sections. In these sections, I discuss a few potential uses of more refined
 measures of legal writing, and then I discuss some potential future studies that would
 develop and improve my methods to yield new insight.
           When beginning my work, my core hypotheses were:
      H1. Writing style differs by court.
      H2. The style at the United States Supreme Court is probably the most distinct, as those briefs
          are produced by some of the most highly respected and highly paid advocates in the country.
      H3. In persuasive briefs, writing style matters. Content may be queen, but style is at least a
           princess.
 To investigate these hypotheses, I examined three courts of review—two final and one
 intermediate. Briefs are easy to gather for these courts, cases are randomly assigned to
 panels, and the overall load is small, meaning I could measure a large percentage of the
 workload.107 I made a few other decisions to narrow and refine the work, which I offer as
 caveats here. Specifically, because my expertise lies in the civil realm, I focused the work
 on civil matters. I deleted cases with cross-appeals because they do not always produce
 a clear winner and loser, and I pulled only appellant opening briefs, scoring a reversal as
 a win. I selected 600 cases (200 from the U.S. Supreme Court, 200 from the Ninth Circuit
 Court of Appeals, and 200 from the California Supreme Court). To select the cases, I took
 the last 200 cases decided. I obtained the briefs in Word format, removed the materials
 that were not actual content meat (style, table of authorities, signature block), and scored
 the writing style of the brief ’s author.
              To score the briefs, I needed a program that would produce meaningful, rich
 data. I tried existing tools in Word, but the metrics they produced were not refined
 enough for statistical work. For example, grade level was measured only to the whole
 grade (as opposed to the tenth or hundredth), making detailed differentiation difficult.
 The program had no settings for legal briefs, and it dealt poorly with things like citation.
 When I ran a first statistical analysis, it became clear that identifying meaningful
107
      The cases, information and measures are included in the Appendix.


                                                      312
 2020]                            UNIVERSITY OF BOLOGNA LAW REVIEW                                      [Vol. 5:2


 differences would require data that could measure more aspects of writing and with
 more precision.         Similarly, the reading level tests were overly simplistic.108                       After
 studying a variety of options, I settled on an innovative program called StyleWriter that
 measures specific writing characteristics, including:
   a. average sentence—average number of words per sentence;
   b. passive index—the percentage of sentences that contain passive verbs;109
   c. style index—how well sentences are written as a whole, with a lower score indicating
           a better sentence;110
      d.   bog index—how easy sentences are to read;111
      e.   sentence bog—the length of the sentences;112
      f.   word bog—the difficulty of words;113
      g.   reading grade—the program eschews the familiar and relatively basic
           Flesh-Kincaid reading score and replaces it with a measure that considers the
         difficulty of the vocabulary, sentence length, and more;
      h. jargon— overused and unnecessarily complicated terms, sometimes called
           “legalese”;114
108
    In a pilot study, I selected one hundred briefs and scored them in Word. I then did some back of the envelope
    investigation about whether the measures might correlate with outcomes. I found no obvious correlations.
109
    See New Features Guide: StyleWriter-4, Editor Software 1, 11-12 (2020), https://www.editorsoftware.com/image
    s/StyleWriter/StyleWriter4_New_Features.pdf. According to StyleWriter, the passive index measures one
    of the most common style faults in writing – overusing passive verbs. StyleWriter counts the number of
    passive verbs, divides them by the number of sentences and multiplies the result by 100 to give a Passive
    Index. Passive Index = Number of Passive VerbsNumber of Sentences × 100.
110
    Editor Software, What is the Style Index?, StyleWriter-4 Support (last visited February 7, 2020),
    http://www.editorsoftware.com/Faqs.html# (follow “Support” hyperlink; then follow the “What is the Style
    Index” hyperlink). StyleWriter indicates that “style” is a dated measure in its program. Id. It suggests that
    “Bog” is now a more complete measure. Id. For that reason, I’ve noted “style” but do not dwell on it. Here is
    StyleWriter’s explanation:
                  The Style Index was StyleWriter’s measure of good writing before we designed the
                  Bog Index. The Style Index measures all plain English problems in the text, including
                  a weighted score for long sentences. It then converts this measure into an index. The
                  best writing consistently scores below 20 – equivalent to two style faults for every
                  100 words. As the Bog Index also measures the plain English problems in the text,
                  we recommend you use the Bog Index.
111
    See New Features Guide, supra note 109, at 15. According to StyleWriter, the Bog Index has three distinct parts,
    (1) Sentence Bog, (2) Word Bog, and (3) Pep. Bog is anything that detracts from easy reading (i.e. bogs
    a reader down). Id. Pep is anything that makes writing easier to read and more interesting (i.e. peps up
    writing). Id. Bog Index = [Sentence Bog + Word Bog – Pep].
112
    A better readability formula: StyleWriter’s Bog Index, StyleWriter - USA (last visited Feb. 7, 2020),
    http://www.stylewriter-usa.com/stylewriter-editing-readability.php. [hereinafter Style Writer’s Bog Index].
    Sentence bog deals with the problem of sentence length. StyleWriter take the Average Sentence Length for
    the document, squares it, then divides the result by the Long Sentence Limit for the chosen Writing Task.
    This reflects the fact that some Writing Tasks demand shorter sentences. Sentence Bog = Average Sentence
    Length2Long Sentence Limit.
113
    See New Features Guide, supra note 109, at 20. Word Bog is the measure of word difficulty. StyleWriter’s
    Bog Index measures: (1) word difficulty, (2) abbreviations and acronyms, (3) wordiness, (4) passive verbs,
    and (5) style issues. In measuring these factors, the program assigns a Bog value to each of these and
    expresses the result as the amount found in 250 words of the document. Word Bog = Style Problems +Heavy
    Words+Abbreviations+Specialist × 250 Number of words.
114
    Id. at 10. The StyleWriter program highlights three forms of jargon: (1) abbreviations and acronyms, (2)
    difficult words outside the understanding of most readers, and (3) jargon phrases.
                                                        313
                                              EX MACHINA


         i. glue—how well the sentences are pulled together;115 and

         j. pep—anything that makes writing easier to read and more interesting.116



 The Bog Index is a better measure of readability because it captures the plain English
 attributes of writing (e.g., active voice, clarity), ather than presupposing that all words
 with multiple syllables are complex and less readable, as in computing the readability
 through Microsoft Word. The word familiarity used by StyleWriter determines the word
 complexity and is based on a wide lexicon of 200,000 words. Thus, the Bog Index measure
 of writing clarity overcomes the major criticism of other readability programs related to
 apprehending readability and complexity founded on syllable totals or passive verbs.
               Besides these measures, I captured data regarding average word length and
 paragraph length. And to avoid confusion by citation or legal terminology, I set the
 program to “legal.” The program still scored citation, but it did not count citation errors
 as such (or as a set of errors). Similarly, the legal setting did not flag all legal words as
 overly complicated, since many are terms of art. Admittedly, the setting was imperfect,
 but they were a marked improvement over many alternatives. To the extent it might still
 score some legal work poorly, those effects will probably be similar across the samples.
 Although the measures may contain errors compared to a hypothetical perfect scoring
 system for briefs, the relative measures between briefs are accurate.
               Figure 1 is an image of how the output from a document appears in StyleWriter.
 The left side shows a visual depiction of a variety of measures. The right side shows notes,
 comments, and suggestions. The bottom left contains a measure of sentence length, along
 with distributing various sentence lengths. The very bottom shows the final raw scores
 for a variety of cumulative measures.




115
      Id. at 22.
               Glue words are the 200 or so most common words in the English language (excluding
               personal pronouns). They are necessary to link nouns, verbs, adverbs and adjectives
               in any sentence. Most writers use too many glue words and almost every document
               could benefit from running an editorial pen through unnecessary glue words.
116
    See Style Writer’s Bog Index, supra note 112. Pep counts the features in the document that are the
    hallmarks of good writing. Pep reduces the overall Bog Index because it can ease the job of the reader
    and make the writing more enjoyable to read. Pep includes the following: names, interesting words,
    conversational expressions, personal pronouns, contractions, direct questions, and variation in sentence
    length. The program assigns a Pep value to each of the features listed and expresses the result as
    the amount found in 25 words of the document (1/10 of the effect from Bog). Pep = Names+Interest
    Words+Conversational×25Number of words+Sentence Variety Sentence Variety = Standard Deviation × 10
    Average Sentence Length.


                                                    314
 2020]                              UNIVERSITY OF BOLOGNA LAW REVIEW                                  [Vol. 5:2




              Figure 1: StyleWriter 4, Standard Edition developed by Editor Software Ltd (UK)



              I recorded the data from each brief and analyzed it to determine whether: (1) the
 writing style in briefs differed by court; and, (2) any specific characteristic or combination
 of characteristics correlated at a statistically significant level with higher win rates. The
 data was analyzed using a variety of statistical measures and techniques.117
              The results are discussed below.

117
      In determining whether courts were similar or different, we determined whether means and variances
      of the writing measures differ between pairs of courts in the three court data. The core steps required
      were: perform exploratory data analysis, plot densities of the writing measures for each court, assess
      the normality of the distributions of the measures in each court from the plots and using the Shapiro-
      Francia test, examine the correlations of the measures, compare the variances of the writing measures
      between pairs of courts, perform pairwise Brown-Forsythe tests on the measures between the courts as
      a non-parametric test of equality of variance, and report sample standard deviations to show direction and
      size of differences. We also examined whether writing characteristic correlated with positive outcomes by
      fitting logistic regression models of the case outcome on the writing measures of the brief. This required
      fitting models with each measure and the court identifier as predictive variables, examining the magnitude,
      direction, and significance of the association of the writing measure with the effect, identifying the best
      subset of predictors according to the A.I.C criterion, examining the predictive power of this model, and
      using a permutation test to assess the significance of the model.


                                                        315
                                                    EX MACHINA


        3.1. UNITED STATES SUPREME COURT BRIEFS ARE SIMPLER

 In this section, I have discussed my findings regarding how briefs compare across the
 selected courts of review. My fundamental findings challenge the existing notion that
 briefs are about the same is inaccurate.                    With a more refined measure, small, but
 statistically significant differences can be identified. This confirms my first hypothesis.
              In a previous work, Long and Christensen concluded, using basic readability
 statistics, that briefs are all about the same.118 That is not true when the tools are
 refined. Briefs differ at a statistically significant level from court to court, as shown in
 the charts below.
              The differences show up in almost every style measure. The charts, Figures 2, 3,
 and 4 below demonstrate the differences.




                                                        Figure 2




118
      See generally Long & Christensen, supra note 74, at 147.


                                                          316
 2020]                              UNIVERSITY OF BOLOGNA LAW REVIEW                                     [Vol. 5:2




                                                       Figure 3




                                                       Figure 4



 These are merely descriptive statistics, and the differences are, sometimes, small. For
 example, the difference in reading level between the United States Supreme Court and
 the California Supreme Court is only 0.60. To investigate whether these differences are
 statistically significant, a T-test was conducted.119 The results are displayed in the table
 below.
119
      A t-test is a type of inferential statistic used to determine if there is a significant difference between the
      means of two groups, which may be related in certain features. A t-test is used as a hypothesis testing tool,
      which allows testing of an assumption applicable to a population.


                                                          317
                                                           EX MACHINA


                                      Average scores with pairwise t-tests of equality of the means
      Measure            California   Ninth     SCOTUS     California v. Ninth   California v. United States   Ninth v. United States
      Total words         8770.97     7334.07   12111.49         0.0000                    0.0000                     0.0000
      Average sentence      16.72       16.47     14.97          0.3926                    0.0000                     0.0000
      Passive index         20.86       19.94     15.22          0.1663                    0.0000                     0.0000
      Style index           83.97       79.89     79.73          0.0069                    0.0034                     0.9117
      Bog index             70.17       70.38     67.67          0.8436                    0.0087                     0.0096
      Reading grade         12.82       12.58     12.18          0.1021                    0.0000                     0.0020
      Jargon               0.0338      0.0404    0.0390          0.0000                    0.0001                     0.3648
      Glue                 0.4036      0.3934    0.3983          0.0001                    0.0211                     0.0289
      Pep                12.21827       12.50     12.96          0.1197                    0.0000                     0.0039

                                                Table 1: Comparing the court


 The average scores for each court are reported in the first three columns. The lower the
 number, the simpler the brief. For example, the Bog Index score for the United States
 Supreme Court is 67.67, almost three points lower than the scores of the other courts. The
 light blue boxes indicate the simplest briefs, and the darker the box, the more complex the
 briefs. A visual inspection shows that the U.S. Supreme Court briefs are the least complex
 in most categories, and briefs in the California Supreme Court are the most complex. For
 the statistically minded, the last three “t-test” columns show whether the differences are
 statistically significant. The darker the green, the more significant the differences. White
 boxes indicate no significance. Scores below .05 are statistically significant. For example,
 in the far right column, the scores for the United States Supreme Court are compared to
 those of the Ninth Circuit. Almost all scores are far below .05. The differences between the
 U.S. Supreme Court briefs, when compared with the Ninth Circuit and California Supreme
 Court, are statistically significant for the majority of categories. The differences between
 the California Supreme Court and the Ninth Circuit are not statistically significant for most
 categories.
                The U.S. Supreme Court briefs were, on the whole, simpler and clearer. Average
 sentence, passive index, bog index, style index, and reading grade120 registered the
 simplest (lowest) scores. The briefs also scored best (highest) on pep, and were in the
 middle on jargon and glue. This confirms my second hypothesis.
                The fact that the Supreme Court briefs are simpler might be surprising. The U.S.
 Supreme Court has a light caseload, the Court hand-picks the cases to hear, each justice has
 four clerks to assist them, the issues presented are all important, they are briefed by some
 of the largest firms in the country, and the average attorney has decades of experience. We
 might expect this to produce an advanced diction, complex sentences, and more focus on
120
      The reading level is driven down by citations. In experimenting with the program, I found that if I deleted
      citations and scored the same text, I often saw a grade level increase in reading level. Many other scores,
      including bog (considered a more complete measure of writing by StyleWriter) remained roughly the same.
      As such, it is possible that the reading grade level is driven down in the United States Supreme Court by
      heavy citation.


                                                                 318
 2020]                              UNIVERSITY OF BOLOGNA LAW REVIEW                                    [Vol. 5:2


 content than on style. The Court has plenty of time to discern the meaning of the briefs,
 and the issues themselves are complex. Instead, we see the opposite: style matters, and
 simplicity and clarity are the norm.



        3.2. WRITING STYLE CORRELATES TO WINNING

 But do simpler briefs win more? This answer is a little more difficult to discern from the
 data, but there are indications that it correlates significantly with winning. The scores
 from all briefs in all courts were considered, no single writing measure was a significant
 predictor of a successful outcome for the appellant. Statisticians typically want to see a
 “p-value” of less than 0.05. This can require a massive sample, as a “p-value” reflects both
 how much of a difference variables make and the number of data points. But, based on the
 briefs cases studied, there is trending evidence that good writing correlates to winning.
              Table 2 shows the coefficient of each writing measure and its “p-value”. A positive
 coefficient means that a higher score is associated with a greater probability of winning.
 Conversely, a negative coefficient indicates that a lower value of the measure is associated
 with a greater probability of winning. So, we would expect to see positive numbers for
 glue and pep (all indicia of good, clear writing), and lower scores for jargon, passive index,
 style, reading grade, and bog (all indicia of muddled, boring, or confusing writing). And
 this is just what we see.121

              Summary of regressions with court indicator and single writing measure
              Measure           Coefficient                   p-value
              Average sentence     0.0056                     0.8669
              Passive index       -0.0182                     0.2031
              Bog index           -0.0133                     0.1179
              Reading grade       -0.0607                     0.3565
              Jargon              -6.8579                     0.2457
              Glue                 3.1202                     0.3919
              Pep                  0.0745                     0.1490

                                                       Table 2



 Here, we see that lower passive index scores, bog index scores, jargon, and reading level
 all correlate to better outcomes. The results are not statistically significant, but the “p-
 values” are much lower, suggesting a larger sample might yield statistically significant

121
      Average sentence length is a variable that doesn’t fully fit my predictions. Longer sentences correlated with
      winning, though not at a statistically significant level.



                                                         319
                                                   EX MACHINA


 results.122 We also find that glue and pep correlate with positive outcomes, again with
 relatively low “p-values”. The finding that does not fit the hypothesis is the measure of
 sentence length. It suggests that longer sentences correlate with winning, but the results
 produce a tremendously high “p-value”, suggesting this result is likely just noise in the
 data. The data suggests the fact that good writing matters, but it is far from conclusive.
            To further analyse, I hypothesized that, if writing does matters, it probably
 matters more in intermediate courts where the workload is higher, the issues are more
 mundane, the review is mandatory, the judges have less help from clerks, and on
 average, more cases are affirmed. There, the quality of writing style might have a
 significant impact on whether the brief is well received and understood because these
 courts aren’t likely to have the time to dig through convoluted writing to uncover the
 deeper meaning. If this hypothesis is right, of the courts I studied, style should matter
 most in the Ninth Circuit.
            And it did as in the Ninth Circuit, a low passive index correlated significantly to
 win rate, with a low bog index coming close to statistical significance.123 This means that,
 in the Ninth Circuit, if you knew only the scores for passivity and bog, you could predict
 whether a case would be reversed or affirmed at a rate a little better than chance.124 Being
 able to predict case outcomes on so little data is perhaps surprising, given one would know
 nothing of the issues, the firm, the attorney, the panel the case was assigned to, the quality
 of the content, the framing, or how oral argument went. Many might predict that without
 measuring content, or lawyer skill, prediction is impossible. But the data does not suggest
 that is true.




122
    P-values help decide if an effect is statistically significant. The smaller the difference observed, the larger
    the sample must be to be sure that the effect is real. For example, if you flip a coin 100 times, and get 51
    heads and 49 tails, you cannot conclude the coin is unfair and slightly favors heads. The difference in the
    results is too small. But if you flipped it 10,000 times, and had 5,100 heads and 4,900 tails, the effect is likely
    real. The coin is probably imbalanced and favors heads. A p-value quantifies this idea, by considering the
    effect and the size of the sample. Here, in the data we see relatively low p-values, consistent with our other
    results. It is very possible that if we increased our sample, the differences we measured would persist, and
    they would test as statistically significant.
123
    I note here that at least one result across courts that I discovered was a bit confusing, at first glance. I found
    that overall, a higher style score (meaning the style is not good), correlated with winning. That is at odds
    with my overall findings. However, upon investigation, I learned that StyleWriter largely moved away from
    the style score, viewing the Bog index score as more complete. It kept the number because clients were used
    to it, but explained the Bog index score was a far better measure. The style score, it appears, may be driven
    higher by citation in legal writing, making it an unreliable measure for this study.
124
    The model only improves on chance by about 2%. But that is not surprising here, as there are large number
    of other potential explanatory variables that have not been coded or analyzed. In future work, with more
    detailed analysis, a combination of content analysis, style analysis, and consideration of other factors could
    combine to produce tools that successfully predict outcomes at a rate far in excess of chance. As discussed
    in this article, that has far reaching implications for a variety of industries.


                                                          320
 2020]                              UNIVERSITY OF BOLOGNA LAW REVIEW                                  [Vol. 5:2


                                           Ninth Circuit Analysis
                                   Court coefficent Estimate p-value
                                   Passive index      -0.0733     0.0079
                                   Bog index          -0.0257     0.1226

                                                      Table 3


 These findings suggest that writing style matters. Indeed, style alone can be used to
 predict outcomes – suggesting it either matters independently or that it somehow
 correlates heavily with other factors that drive resolution. If writing matters in the
 Ninth Circuit, it might be expected to matter even more in trial courts, where judges
 have more work, less time to dive into issues, and are therefore more reliant on briefs
 and the explanations provided by parties. I plan to test this hypothesis in a future study.
 That data will reveal whether, as court workloads increase and the issues become less
 earth-shattering, the importance of writing style increases.




          4. IMPROVING THE METHOD AND TOOLS AVAILABLE

 My early results suggest that precise measures of legal writing style are possible. The
 data also suggests that when we measure legal writing precisely, the style of legal writing
 can predict outcomes.125 But, understanding whether legal writing style correlates with
 outcomes, or has a causal effect on them, requires better tools and replication.
              With regard to refined tools, one could imagine a program developed specifically
 to measure legal writing. I hijacked a tool designed to make sure writing is clear and
 simple, however, a tool specifically designed to measure legal writing could identify
 citations, and, either exclude them from the measures, and/or count them effectively, as
 another metric to use when considering how writing style relates to outcomes. A
 legal-writing-specific analysis tool could identify legalese and differentiate it from terms
 of art. For example, it might treat “heretofore” as legalese, while viewing “proximate
 cause” as a term of art. This tool could be adjusted to consider elements like the
 frequency of headings, overall length of sections, whether an introduction is included,
 whether the writing contains intensifiers, and much more.                             This would provide
 additional real measures to potentially gain new insights into legal writing and
 persuasion.
125
      As noted supra, the most powerful predictive software will marry measures of style with measures of content
      and other factors.


                                                        321
                                                   EX MACHINA


 Beyond a more refined tool to measure legal writing, future studies (mine and others) need
 to code for more information both to identify other explanatory variables and to more
 fully consider correlation versus causation. For example, I did not code my data for firm
 size, years of practice experience, past appellate wins, or several other characteristics that
 could explain the outcomes. And I did not measure legal content, which might be possible
 using some of the software discussed earlier in this article. Doing so would provide a whole
 new round of descriptive statistics and provide new answers to interesting questions. Do
 solo practitioners write differently than big firm lawyers? Do appellate attorneys with
 more years of experience change how they write? Does quality of content correlate with
 quality of style? And so on.
              Similarly, one could imagine that what matters most in persuasive writing might
 be the delta between the two briefs. The process is adversarial, so maybe writing style
 matters most when there are pronounced differences between the two sides?126 Or maybe
 writing style matters because it works better when it correlates with the “house style” of
 the court reading it. Or maybe, writing style is a proxy of sorts – a signal that correlates
 with more time spent on briefs, or more careful research, or more attorneys to work on
 the brief.
              This merits further exploration. I hope the growing ranks of empirical legal
 researchers will, occasionally, focus their powerful tools on persuasive legal writing.
 And, as discussed below, I am virtually certain that even if they do not, venture capital
 will. There is a massive untapped market in legal writing.




          5. IMPLICATIONS FOR THE PROFESSION IN THE FUTURE

 Evolution in the ability to measure persuasive legal writing effectively and to understand
 its impact on outcomes will have broad impacts in a variety of fields, including software
 development for lawyers, insurance, legal finance, and the teaching of legal writing. I
 briefly discuss each below.



126
      In my first round, I scored the briefs of both sides for 200 briefs. We measured the delta between various
      writing measures and looked for any patterns or evidence that differences in style explained outcomes. We
      found no such evidence, but the work was preliminary and would benefit from further innovation. It is also
      possible that such differences would be far more pronounced in trial courts where there is likely to be both
      more variation and more importance placed on easy-to-read briefs, given the workloads of many courts and
      the relative lack of help. For example, in many state courts, judges have no clerks and make decisions on
      their ownunder significant time pressure.


                                                         322
 2020]                            UNIVERSITY OF BOLOGNA LAW REVIEW                                [Vol. 5:2


 Software Development: As tools for measuring persuasive legal writing are developed, as we
 understand how those measures relate to outcomes, text analysis software will become
 common place. If data reveals certain characteristics of persuasive writing cause better
 outcomes, it will become malpractice to fail to measure an attorney’s work against the
 identified thresholds. Just as many companies inspect all written text they plan to publish
 with StyleWriter before they allow it to be released, one can imagine a day when law firms
 demand all briefs are “scored” using software before the briefs are submitted.
              Small firms may buy software. Even today, a firm I am familiar with regularly
 uses a style software to measure briefs before they are filed. Large firms may well hire
 programmers and people trained in empirical methods to develop proprietary, in-house
 software. In a competitive legal market where big firms often compete with one another
 for clients, marketing that all persuasive legal writing is refined with proprietary software
 proven to improve results may produce a real edge.
              The best software will not be static, either. It will deploy machine learning, a form
 of A.I., to constantly improve and update. A sophisticated, large firm could score all briefs,
 and then require that attorneys enter the outcome when the judge rules. The software
 would, over time, refine its algorithms, allowing it to provide evolving “advice”. In large
 enough firms, with adequate time, such advice might even differ by court or by judge.
              Insurance: Insurance for verdicts is a growing sector. When an attorney in a civil
 case obtains a large verdict, companies often approach the firm and offer the
 opportunity to insure the verdict. For example, if a plaintiff obtains a $10 million verdict,
 the company might offer to insure the verdict for $2 million. The plaintiff pays $2
 million, and for that, they are guaranteed even if they lose, they receive $10 million.127
 This insurance works like all insurance – the company is estimating a claim, and then
 pricing across a book of cases. But how the insurance company scores reversal is harder
 to know. At a minimum, it involves evaluating the track record of the attorneys, the
 strength of the legal positions, the makeup of the reviewing court, the reversal rates
 from that court and more. The insurance companies could wait to offer insurance until
 opening and appellate briefs are filed if they wished, and gain an additional data point by
 scoring those briefs.           Or, the insurance company could insure, but only upon a
 requirement that the submitted briefs are scored and pass various benchmarks shown to
 correlate, or partially cause, better outcomes.
              Legal Financing: The number of businesses, attorneys, and banks willing to invest
 money in cases in exchange for a generous return on the investment is increasing and

127
      Some insurance companies I’ve encountered have far more complex formulas, for example providing sliding
      percentages of guaranteed recovery based on the premium. But I offer the example above for simplicity.


                                                      323
                                                   EX MACHINA


 becoming mainstream. Websites like lexshare.com allow anyone to read about a case,
 investigate its strength, and then invest in exchange for a promised return if it settles or
 results in a favorable verdict.128 Similarly, several funds have been established on Wall
 Street that invest in cases, and the American Bar Association has documented the growth
 of legal finance companies.129 Those companies operate in several ways, but the two
 principal ways are: (a) to monetize existing legal assets (such as paying money to a
 company now based on its pending legal cases), or (b) loaning money directly to lawyers
 to fund their ongoing litigation (in which case the loans are often secured by a book of
 the lawyer’s existing business).130
            Companies in legal finance require detailed information to evaluate investments,
 and they typically require various forms of routine and regular updates on litigation.
 They hire lawyers and others to evaluate risk and to decide when and how to invest.131
 Much like the insurance section, software that scored persuasive legal writing and
 improved outcomes would be invaluable. Massive funds like Burford Capital, a legal
 finance company that has $3.3 billion to invest in legal matters, could develop its own
 software as a proactive way to improve its returns.132 Or, it could at a minimum use
 existing software to score the work of firms and cases it invests in, including requiring all
 briefs filed to meet certain benchmarks.
            Teaching Legal Writing: Perhaps the most promising innovations from improved
 measure and analysis of legal writing will occur in the teaching of legal writing. For
 centuries, legal writing has been taught based on hunches, personal experience, and
 instincts. Some schools relegate legal writing to adjuncts, who may be competent
 writers, but who may not have the time or training to consider the science of writing.
 Still others allow students to teach the course. Outside the academy, people like Bryan
 Garner have made millions of dollars selling legal writing advice based on their informed
 guesses about what works. And law professors, including me, often teach students the
 “truths” of legal writing. But I have learned when sitting in meetings and conferences
 with other legal writing teachers that my “truths” not always align with the truths of

128
    See, e.g., LexShares, https://www.lexshares.com/cases (last visited Feb. 14, 2020).
129
    See also Mary E. Egan, Other People’s Money: Rise of Litigation Finance Companies Raises Legal and Ethical Concerns,
    ABA J. (Dec. 1, 2018), https://www.abajournal.com/magazine/article/litigation_ finance_legal_ethical_con
    cerns.
130
    Id.
131
    See, e.g., Pravati Capital, https://pravaticapital.com/litigation-funding-services/ (last visited February 14,
    2020) (“At Pravati Capital, the key to our success is in identifying cases that we know have a great chance at
    winning. When a large or small law firm brings its high probability cases to Pravati Capital, and our team of
    expert underwriters verifies precedent and likelihood of success, we invest in the case—and the firm—with
    a non-recourse cash advance or line of credit against the anticipated settlement.”).
132
    Brian Baker, In Low-yield Environment, Litigation Finance Booms, MarketWatch (Aug.                      21, 2018),
    https://www.marketwatch.com/story/in-low-yield-environment-litigation-finance-booms-2018-08-17.


                                                          324
2020]                     UNIVERSITY OF BOLOGNA LAW REVIEW                         [Vol. 5:2


others, and vice versa. The same is true when I co-counsel with others on appellate
briefs, or when I attend conferences for appellate lawyers. We all think we know what
works – but we cannot all be right.
        Data can clear this fog. With enough studies, we can know what works, and teach it.
This is true for all teachers. Senior partners, people like Garner, and law professors would
benefit immensely from empirical measures and analyses of persuasive legal writing. It
would move the teaching from the twentieth century (to be generous) into the twenty-first
century.
        One could also imagine using advanced legal writing software as one method of
“scoring” student work. Instead of deploying teaching assistants to check grammar and
citations, sophisticated textual analysis software would do the work in seconds and
produce a detailed report for the student. Although this technology could certainly
measure writing style, one could also imagine a future in which it measures both style
and content, producing a detailed report of which cases were cited, how that compares
to the class as a whole or a model brief, along with a variety of measures of the style.
        The natural outflow of this would be that students would adopt the software to
improve their writing. Microsoft Word checks spelling and grammar. Many of us could not
live without those red and blue lines. And students deploy Word to improve their work.
The same will almost certainly become true with advanced software that measures legal
writing. Students will deploy it to improve their own writing, making their work faster
and less tedious, while improving its overall content. They could even compare their style
to the leading styles in high courts, or the courts of their state. This would increase the
quality of the work rapidly, and free up time to talk about more complex issues that cannot,
at least to date, be automated.




    CONCLUSION

In ten years, a legal writing textbook will not be filled with quotes from judges or
anecdotes by lawyers. It will illuminate readers with charts and graphs and data, and in
doing so, it will move the way we think about legal writing away from intuition and
towards irrefutable findings.     The continued evolution of legal research tools, the
overlapping use of textual analysis tools to measure brief content, and a new set of tools
to measure writing style will merge to produce new insights.




                                            325
                                       EX MACHINA


Efforts to study effective legal writing style remain in their infancy, but that is changing
fast. In the next ten years, the measure of persuasive legal writing style will become
more precise, more powerful, and more predictive.           As it does, long unanswered
questions about the role of style, and the ideal approach to persuasive legal writing will
emerge. At first, those may be general answers. But as technology improves, and
researchers and firms alike begin to measure thousands upon thousands of briefs and
track them against outcomes, the lessons will slowly become more granular.             This
progress will be accelerated by A.I., and in particular, machine learning. Briefs will be
analyzed, and results tracked. The machine will learn. It will refine its suggestions for
writing, and lawyers will adapt. And as they do, the feedback cycle will accelerate. Those
of us invested in creating legal writing, and in teaching it, should embrace the change. It
is an opportunity to challenge our own assumptions, and in doing so, become better.




                                            326
