python requests invalid schema error
<p>I am trying to scrape from an online corpus of texts. These texts are arranged in a tree-like fashion on the site: One clicks A which opens a B page, and in B, one clicks C, and it opens the text.   In A there are about ~50 links, in B, it varies between 3 and ~150, there are also sometimes links in C, but I am not interested in them.<br>
Here is what I did to achieve this: I opened the A, I parsed it with BeautifulSoup, I collected the links I wanted, and saved it as a .txt file. Then I did the following:</p>

<pre><code>Url_List=[]

with open("Aramaic_Url_List.txt", "r") as Url_List:
    urls=Url_List.read()

A_url_list=urls.splitlines()  
Yeni_A_url_list=[showsubtexts for showsubtexts in A_url_list if len(showsubtexts)&gt;52]  
</code></pre>

<p>Which gave me all the links I wanted from page A in a list form.
Then I wrote a small script to test whether I can obtain the links in B page from an element of the list <code>Yeni_A_url_list</code>, here is my script:</p>

<pre><code>data2=requests.get(Yeni_A_url_list[1].strip())
data2.raise_for_status()
data2_Metin=data2.text

soup_data2=BeautifulSoup(data2_Metin, "lxml")

for link in soup_data2.find_all("a"):
    print(link.get("href"))
</code></pre>

<p>The strip probably has no function there, but I thought it wouldn't hurt. The script worked rather well for an element. So I thought, it is time to write a function to obtain all the links in page B level for every link in page A. So here is my function:</p>

<pre><code>def ListedenLinkAl(h):
    if h in Yeni_A_url_list:
        print(h)
    g=requests.get(h)
    g.raise_for_status()
    data_mtn=g.text
    data_soup=BeautifulSoup(data_mtn,"lxml")
    oP=[b.get("href") for b in data_soup.find_all("a")]
    tk=list(set(oP))
    sleep(3)  
    return tk
</code></pre>

<p>print is there for me to see the links that have been worked out by the function, and sleep is there to not overcharge the server though for some reason time.sleep revealed an error in syntax. The function also worked for a single element of the list, meaning the following worked: <code>ListedenLinkAl(Yeni_A_url_list[1])</code></p>

<p>So I thought, it is time to apply this function to every element of the list <code>Yeni_A_url_list</code> and did a list comprehension:</p>

<pre><code>Temiz_url_Listesi=[ListedenLinkAl(x) for x in Yeni_A_url_list]
</code></pre>

<p>And I received the following error:</p>

<pre><code>In [45]: Temiz_url_Listesi=[ListedenLinkAl(x) for x in Yeni_A_url_list]
http://cal1.cn.huc.edu/showsubtexts.php?keyword=21200
Traceback (most recent call last):

  File "&lt;ipython-input-45-8e4811c83c3f&gt;", line 1, in &lt;module&gt;
    Temiz_url_Listesi=[ListedenLinkAl(x) for x in Yeni_A_url_list]

  File "&lt;ipython-input-45-8e4811c83c3f&gt;", line 1, in &lt;listcomp&gt;
    Temiz_url_Listesi=[ListedenLinkAl(x) for x in Yeni_A_url_list]

  File "&lt;ipython-input-36-390e6ed1eae5&gt;", line 6, in ListedenLinkAl
    g=requests.get(h)

  File "/home/dk/anaconda3/lib/python3.5/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)

  File "/home/dk/anaconda3/lib/python3.5/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)

  File "/home/dk/anaconda3/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)

  File "/home/dk/anaconda3/lib/python3.5/site-packages/requests/sessions.py", line 570, in send
    adapter = self.get_adapter(url=request.url)

  File "/home/dk/anaconda3/lib/python3.5/site-packages/requests/sessions.py", line 644, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)

InvalidSchema: No connection adapters were found for 'http://cal1.cn.huc.edu/showsubtexts.php?keyword=21200'

In [46]: 
</code></pre>

<p>I have no idea why the function works for a single element in the list, but not in the list comprehension.</p>