Scraping a table from a section in Wikipedia
<p>I'm trying to come up with a robust way to scrape the final standings of the NFL teams in each season; wonderfully, there is <a href="https://en.wikipedia.org/wiki/List_of_National_Football_League_seasons" rel="nofollow">a Wikipedia page</a> with links to all this info.</p>

<p>Unfortunately, there is a lot of inconsistency (perhaps to be expected, given the evolution of league structure) in how/where the final standings table is stored.</p>

<p>The saving grace <em>should</em> be that the relevant table is always in a section with the word "Standings".</p>

<p>Is there some way I can <code>grep</code> a section name and only extract the <code>table</code> node(s) there?</p>

<p>Here are some sample pages to demonstrate the structure:</p>

<ul>
<li><p><a href="https://en.wikipedia.org/wiki/1922_NFL_season" rel="nofollow">1922 season</a> - Only one division, one table; table is found under heading "Standings" and has xpath <code>//*[@id="mw-content-text"]/table[2]</code> and CSS selector <code>#mw-content-text &gt; table.wikitable</code>.</p></li>
<li><p><a href="https://en.wikipedia.org/wiki/1950_NFL_season" rel="nofollow">1950 season</a> - Two divisions, two tables; both found under heading "Final standings". First has xpath <code>//*[@id="mw-content-text"]/div[2]/table</code> / CSS <code>#mw-content-text &gt; div:nth-child(20) &gt; table</code>, second has xpath <code>//*[@id="mw-content-text"]/div[3]/table</code> and selector <code>#mw-content-text &gt; div:nth-child(21) &gt; table</code>.</p></li>
<li><p><a href="https://en.wikipedia.org/wiki/2000_NFL_season" rel="nofollow">2000 season</a> - Two conferences, 6 divisions, two tables; both found under heading "Final regular season standings". First has xpath <code>//*[@id="mw-content-text"]/div[2]/table</code> and selector <code>#mw-content-text &gt; div:nth-child(16) &gt; table</code>, second has xpath <code>//*[@id="mw-content-text"]/div[3]/table</code> and selector <code>#mw-content-text &gt; div:nth-child(17) &gt; table</code></p></li>
</ul>

<p>In summary:</p>

<pre><code># season |                                   xpath |                                          css
-------------------------------------------------------------------------------------------------
#   1922 |     //*[@id="mw-content-text"]/table[2] |           #mw-content-text &gt; table.wikitable
#   1950 | //*[@id="mw-content-text"]/div[2]/table | #mw-content-text &gt; div:nth-child(20) &gt; table
#        | //*[@id="mw-content-text"]/div[3]/table | #mw-content-text &gt; div:nth-child(21) &gt; table
#   2000 | //*[@id="mw-content-text"]/div[2]/table | #mw-content-text &gt; div:nth-child(16) &gt; table
#        | //*[@id="mw-content-text"]/div[3]/table | #mw-content-text &gt; div:nth-child(17) &gt; table
</code></pre>

<p>Scraping, e.g., 1922 would be easy:</p>

<pre><code>output &lt;- read_html("https://en.wikipedia.org/wiki/1922_NFL_season") %&gt;%
  html_node(xpath = '//*[@id="mw-content-text"]/table[2]') %&gt;% whatever_else(...)
</code></pre>

<p>But I didn't see any pattern that I could use in the xpath nor the CSS selector that I could use to generalize this so I don't have to make 80 individual scraping exercises.</p>

<p>Is there any robust way to try and scrape all these tables, especially given the crucial insight that all the tables are located below a heading which would return <code>TRUE</code> from <code>grepl("standing", tolower(section_title))</code>? </p>