Web scraping using WebBrowser in a class library
<p>I need to create a method in a class library to get the content of a URL (which may be dynamically populated by JavaScript).</p>

<p>I am clueless, but having googling for the whole day this is what I came up with: (Most of the code is from <a href="https://stackoverflow.com/questions/20930414/how-to-dynamically-generate-html-code-using-nets-webbrowser-or-mshtml-htmldocu/20934538#20934538">here</a>)</p>

<pre><code>using System;
using System.Threading.Tasks;
using System.Threading;
using System.Windows.Forms;

public static class WebScraper
{
    [STAThread]
    public async static Task&lt;string&gt; LoadDynamicPage(string url, CancellationToken token)
    {
        using (WebBrowser webBrowser = new WebBrowser())
        {
            // Navigate and await DocumentCompleted
            var tcs = new TaskCompletionSource&lt;bool&gt;();
            WebBrowserDocumentCompletedEventHandler onDocumentComplete = (s, arg) =&gt; tcs.TrySetResult(true);

            using (token.Register(() =&gt; tcs.TrySetCanceled(), useSynchronizationContext: true))
            {
                webBrowser.DocumentCompleted += onDocumentComplete;
                try
                {
                    webBrowser.Navigate(url);
                    await tcs.Task; // wait for DocumentCompleted
                }
                finally
                {
                    webBrowser.DocumentCompleted -= onDocumentComplete;
                }
            }

            // get the root element
            var documentElement = webBrowser.Document.GetElementsByTagName("html")[0];

            // poll the current HTML for changes asynchronosly
            var html = documentElement.OuterHtml;
            while (true)
            {
                // wait asynchronously, this will throw if cancellation requested
                await Task.Delay(500, token);

                // continue polling if the WebBrowser is still busy
                if (webBrowser.IsBusy)
                    continue;

                var htmlNow = documentElement.OuterHtml;
                if (html == htmlNow)
                    break; // no changes detected, end the poll loop

                html = htmlNow;
            }

            // consider the page fully rendered 
            token.ThrowIfCancellationRequested();
            return html;
        }
    }
}
</code></pre>

<p>It currently throws this error</p>

<blockquote>
  <p>ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be
  instantiated because the current thread is not in a single-threaded
  apartment.</p>
</blockquote>

<p>Am I close? Is there a fix for the above?</p>

<p>Or if I am off the track, is there a ready solution to get dynamic web content using .NET (that can be called from a class library)?</p>