Is there an easier way to use a dictionary to scrape multiple webpages?
<p>I am trying to scrape yellow pages using requests. I know that a login is not required to get the data on these pages but I am just trying to get practice logging onto a site. </p>

<p>Is there a way to use "s.get()" to grab multiple url's at one time? This is how I currently have my code laid out but it seems like there should be an easier way so that I do not have to write five extra lines of code each time I want to add a new page.</p>

<p>This code works for me but seems too long. </p>

<pre><code>import requests
from bs4 import BeautifulSoup
import requests.cookies

s = requests.Session()

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

url = "https://accounts.yellowpages.com/login?next=https%3A%2F%2Faccounts.yellowpages.com%2Fdialog%2Foauth&amp;client_id=590d26ff-34f1-447e-ace1-97d075dd7421&amp;response_type=code&amp;app_id=WEB&amp;source=ypu_register&amp;vrid=cc9cb936-50d8-493b-83c6-842ec2f068ed&amp;register=true"
r = s.get(url).content
page = s.get(url)
soup = BeautifulSoup(page.content, "lxml")
soup.prettify()

csrf = soup.find("input", value=True)["value"]

USERNAME = 'myusername'
PASSWORD = 'mypassword'

cj = s.cookies
requests.utils.dict_from_cookiejar(cj)

login_data = dict(email=USERNAME, password=PASSWORD, _csrf=csrf)
s.post(url, data=login_data, headers={'Referer': "https://accounts.yellowpages.com/login?next=https%3A%2F%2Faccounts.yellowpages.com%2Fdialog%2Foauth&amp;client_id=590d26ff-34f1-447e-ace1-97d075dd7421&amp;response_type=code&amp;app_id=WEB&amp;source=ypu_login&amp;vrid=63dbd394-afff-4794-aeb0-51dd19957ebc&amp;merge_history=true"})

targeted_page = s.get('http://m.yp.com/search?search_term=restaurants&amp;search_type=category', cookies=cj)

targeted_soup = BeautifulSoup(targeted_page.content, "lxml")

targeted_soup.prettify()

for record in targeted_soup.findAll('div'):
    print(record.text)

targeted_page_2 = s.get('http://www.yellowpages.com/search?search_terms=Gas+Stations&amp;geo_location_terms=Los+Angeles%2C+CA', cookies=cj)

targeted_soup_2 = BeautifulSoup(targeted_page_2.content, "lxml")

targeted_soup_2.prettify()

for data in targeted_soup_2.findAll('div'):
    print(data.text)
</code></pre>

<p>When I try to use a dictionary like this I get a traceback that I don't understand.</p>

<pre><code>import requests
from bs4 import BeautifulSoup
import requests.cookies

s = requests.Session()

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

url = "https://accounts.yellowpages.com/login?next=https%3A%2F%2Faccounts.yellowpages.com%2Fdialog%2Foauth&amp;client_id=590d26ff-34f1-447e-ace1-97d075dd7421&amp;response_type=code&amp;app_id=WEB&amp;source=ypu_register&amp;vrid=cc9cb936-50d8-493b-83c6-842ec2f068ed&amp;register=true"
r = s.get(url).content
page = s.get(url)
soup = BeautifulSoup(page.content, "lxml")
soup.prettify()

csrf = soup.find("input", value=True)["value"]

USERNAME = 'myusername'
PASSWORD = 'mypassword'

login_data = dict(email=USERNAME, password=PASSWORD, _csrf=csrf)
s.post(url, data=login_data, headers={'Referer': "https://accounts.yellowpages.com/login?next=https%3A%2F%2Faccounts.yellowpages.com%2Fdialog%2Foauth&amp;client_id=590d26ff-34f1-447e-ace1-97d075dd7421&amp;response_type=code&amp;app_id=WEB&amp;source=ypu_login&amp;vrid=63dbd394-afff-4794-aeb0-51dd19957ebc&amp;merge_history=true"})

targeted_pages = {'http://m.yp.com/search?search_term=restaurants&amp;search_type=category',
                  'http://www.yellowpages.com/search?search_terms=Gas+Stations&amp;geo_location_terms=Los+Angeles%2C+CA'
                  }
targeted_page = s.get(targeted_pages)

targeted_soup = BeautifulSoup(targeted_page.content, "lxml")

targeted_soup.prettify()

for record in targeted_soup.findAll('div'):
    print(record.text)

targeted_page_2 = s.get('http://www.yellowpages.com/search?search_terms=Gas+Stations&amp;geo_location_terms=Los+Angeles%2C+CA')

targeted_soup_2 = BeautifulSoup(targeted_page_2.content, "lxml")

targeted_soup_2.prettify()
</code></pre>

<p>Error</p>

<pre><code>raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for '{'http://www.yellowpages.com/search?search_terms=Gas+Stations&amp;geo_location_terms=Los+Angeles%2C+CA', 'http://m.yp.com/search?search_term=restaurants&amp;search_type=category'}'
</code></pre>

<p>I am new to python and the requests module but I don't understand why using the dictionary in this format does not work. Thanks for any input.</p>