Moving between pages and scraping as I go with Nightmare
<p>There is a website that contains a page with a list of 25 entries, where each entry is a link to a page containing some information that I need. I want get to the listing page and then:
1) click on link to first entry
2) retrieve all the html
3) click back to the listing page (there is a button for this)
4) repeat for every other listing</p>

<p>I would also like to do this as efficiently as possible which I've been told means leveraging promises. Here's my code sketch, which doesn't work:</p>

<pre><code>var Nightmare = require('nightmare');
var nightmare = Nightmare({ openDevTools: true, show: true })
var Xray = require('x-ray');
var x = Xray();
var resultArr = [];

nightmare
.goto(hidTestURL)
.wait(2500)
.click('input[name="propertySearchOptions:advanced"]') //start navigating to listing page
.wait(2500)
.type('input[name="propertySearchOptions:streetName"]', 'Main')
.wait(2500)
.select('select[name="propertySearchOptions:recordsPerPage"]', '25')
.wait(2500)
.click('input[name="propertySearchOptions:search"]') //at listing page
.wait(2500)
.then(function(){
  nightmare
  .click('a[href^="Property.aspx?prop_id=228645"]') //first entry
  .evaluate(function(){ //retrieve info
    var resultArr = [];
    resultArr.push(document.querySelector('html').innerHTML);
  })
})

nightmare
.click('a[id="propertyHeading_searchResults"]') //return to listing page
.evaluate(function(){
  return resultArr.push(document.querySelector('html').innerHTML); retrieve listing page info to show that it returned.
})
.then(function (resultArr) {
  console.log('resultArr', resultArr);
  x(resultArr[1], 'body@html') //output listing page html
    .write('results.json');
})
</code></pre>

<p>This gets as far as the listing page, and then does not proceed any further. I also tried the same code, but with <code>return nightmare</code> for every use of <code>nightmare</code> except the first one. I'd seen some examples that used <code>return</code>, but when I did this, the code threw an error.</p>

<p>I also tried not including the third <code>nightmare</code> (the one after the blank space), and instead trying to continue the old nightmare instance by going straight to the <code>.click()</code>, but this also threw an error.</p>

<p>I clearly need some help with the syntax and semantics of nightmare, but there is not much documentation online besides an API listing. Does anyone know how I can make this work?</p>