How to get a JSON response from a Google Chrome Selenium Webdriver client?
<p>Currently I have Selenium hooked up to python to scrape a webpage. I found out that the page actually pulls data from a JSON API, and I can get a JSON response as long as I'm logged in to the page.</p>

<p>However, my approach of getting that response into python seems a bit junky; I select text enclosed in <code>&lt;pre&gt;</code> tags and use python's <code>json</code> package to parse the data like so:</p>

<pre><code>import json
from selenium import webdriver

url = 'http://jsonplaceholder.typicode.com/posts/1'
driver = webdriver.Chrome()
driver.get(url)
json_text = driver.find_element_by_css_selector('pre').get_attribute('innerText')
json_response = json.loads(json_text)
</code></pre>

<p>The only reason I need to select within <code>&lt;pre&gt;</code> tags at all is because when JSON appears in Chrome, it comes formatted like this:</p>

<pre><code>&lt;html&gt;
&lt;head&gt;&lt;/head&gt;
&lt;body&gt;
&lt;pre style="word-wrap: break-word; white-space: pre-wrap;"&gt;{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}&lt;/pre&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<p>And the only reason I need to do this inside selenium at all is because I need to be logged into the website in order to get a response. Otherwise I get a 401 and no data.</p>