How to Make Numerous HTTP Requests Using Task Parallel Library
<p>I am new to the .NET framework's Task Parallel Library (and multitasking and multithreading in general). 
From the literature I have read, I should just be able to create a bunch of tasks,
run them and the framework should take care of spawning the necessary number of
threads according to the resources available on the system.
The problem I have is making thousands of HTTP requests in a timely manner.
So this is the code I have.</p>

<pre><code>var taskList = new List&lt;Task&gt;();
foreach(var request in requests)
{
     taskList.Add(client.SendAsync(request));
}
Task.WaitAll(taskList.ToArray());
</code></pre>

<p><code>client</code> is a <code>System.Net.Http.HttpClient</code> object.
I am using <code>Task.WaitAll()</code> because this code is inside a method that is not async.
To test this code, I am making the requests to another server on the same LAN.
The requests collection has over 15,000 so a task should be created for each object.
But it only manages to run about 7,000 before throwing an aggregate exception. The inner exception doesn't seem to be very helpful, stating only
"A task was cancelled"
though the cancellation token reports that no cancellation was requested.
The stack trace isn't very helpful either with the most recent calls shown being:</p>

<blockquote>
  <p>at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32
  millisecondsTimeout, CancellationToken cancellationToken)
  at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32
  millisecondsTimeout)<br>
  at System.Threading.Tasks.Task.WaitAll(Task[] tasks)</p>
</blockquote>

<p>I also played around with <code>Parallel.Invoke()</code> but that proved much worse.</p>

<pre><code>var taskActionList = new List&lt;Action&gt;();
foreach(var request in requests)
{
    taskActionList.Add(() =&gt; client.SendAsync(request));
}
Parallel.Invoke(taskActionList.ToArray());
</code></pre>

<p>This does not throw any exceptions but it only runs about 1,300 tasks and 
the code runs to completion.</p>

<p>My question is, how do you use the Task Parallel Library to efficiently make 
a large number of HTTP requests? Is there something I am missing?</p>