BeautifulSoup4 performance
<p>Here is simple piece of code to find elements with certain ids. As example I took big random Wiki article.</p>

<p>Code to test:</p>

<pre><code># coding: utf8

from bs4 import BeautifulSoup, Tag
import requests
import time
import sys

TAG_NAME = "li"


def find_with_index(index, id):
    if id and id in index:
        return index[id]
    return None

page_text = requests.get("https://en.wikipedia.org/wiki/United_States").text
page = BeautifulSoup(page_text, 'lxml')
if page:
    print("Page was downloaded and parsed")
else:
    print("Something wrong.")
    sys.exit()

all_ids = set()
for child in page.recursiveChildGenerator():
    if type(child) is Tag and child.has_attr("id") and child.name == TAG_NAME:
        all_ids.add(child.attrs["id"])

print(str(len(all_ids)) + " ids in total")

bs_find_start = time.clock()
[page.find(TAG_NAME, {"id": id}) for id in all_ids]
bs_find_end = time.clock()

index_find_start = time.clock()
simple_index = {li.attrs["id"]: li for li in page.find_all(TAG_NAME) if li.has_attr("id")}
[find_with_index(simple_index, id) for id in all_ids]
index_find_end = time.clock()

print("Spent on bs.find: " + str(bs_find_end - bs_find_start))
print("Spent on indexed find: " + str(index_find_end - index_find_start))
</code></pre>

<p>I have got this output:</p>

<pre><code>Spent on bs.find: 122.81345616345673
Spent on indexed find: 0.027779648046461602
</code></pre>

<p>Here is the question:
this is absolute disaster in terms of performance. Does it mean that BS does not have any sort of indexes inside and will perform find operations by traversing the whole DOM tree again and again no matter what I need to find? Or I do not fully understand how to perform find operations efficiently? When there are a lot of find operations (100+) this may be a severe bottleneck and I can not say that it is very obvious to find the problem.</p>