Access next sibling <li> element with BeautifulSoup
<p>I am completely new to web parsing with Python/BeautifulSoup. I have an HTML that has (part of) the code as follows:</p>

<pre><code>&lt;div id="pages"&gt;
    &lt;ul&gt;
        &lt;li class="active"&gt;&lt;a href="example.com"&gt;Example&lt;/a&gt;&lt;/li&gt;
        &lt;li&gt;&lt;a href="example.com"&gt;Example&lt;/a&gt;&lt;/li&gt;
        &lt;li&gt;&lt;a href="example1.com"&gt;Example 1&lt;/a&gt;&lt;/li&gt;
        &lt;li&gt;&lt;a href="example2.com"&gt;Example 2&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
&lt;/div&gt;
</code></pre>

<p>I have to visit <strong>each link</strong> (basically each <code>&lt;li&gt;</code> element) until there are no more <code>&lt;li&gt;</code> tags present. Each time a link is clicked, its corresponding <code>&lt;li&gt;</code> element gets class as 'active'. My code is:</p>

<pre><code>from bs4 import BeautifulSoup
import urllib2
import re

landingPage = urllib2.urlopen('somepage.com').read()
soup = BeautifulSoup(landingPage)

pageList = soup.find("div", {"id": "pages"})

page = pageList.find("li", {"class": "active"})
</code></pre>

<p>This code gives me the first <code>&lt;li&gt;</code> item in the list. My logic is I am keeping on checking if the <code>next_sibling</code> is not None. If it is not None, I am creating an HTTP request to the <code>href</code> attribute of the <code>&lt;a&gt;</code> tag in that sibling <code>&lt;li&gt;</code>. That would get me to the next page, and so on, till there are no more pages. </p>

<p>But I can't figure out how to get the <code>next_sibling</code> of the <code>page</code> variable given above. Is it <code>page.next_sibling.get("href")</code> or something like that? I looked through the documentation, but somehow couldn't find it. Can someone help please? </p>