ASP.Net IIS Worker Process (w3wp.exe *32) utilizing 100 CPU resources in production environment
<p>We recently deployed a new version of our web application to production with some significant changes to a relatively higher traffic page that loads various mortgage products.  Since the deployment, the this products page increased from 7 to 10 seconds to anywhere from 50 seconds to 2 minutes to load.  Also, the CPU utilization is now often maxed due to our IIS worker process (w3wp.exe 32) running on our dedicated production web server.  This unacceptably drastic loss in performance occurs on the Product page only, and not on any other pages.  </p>

<p>I ran DebugDiag.exe on the server during a sustained duration of maxed out CPU Usage.  The culprit was again our Product page.  We have an average of 60 users during normal business hours and during peak times, the maxed out CPU significantly slows down the rest of the web application page loads.  We get the occasional http error code 501 which indicates server overloads &amp; request failures.</p>

<p>I ran a Chrome Dev Tools > Network > Capture and see that (for example) one request to get just the products page itself (248k) takes 24 seconds, then several subsequent requests for a couple of 20kb Images and JavaScript Files takes even longer (24 to 50 seconds each) for a total page load time of 1.9 minutes (850k transferred, 93 total requests, DOMContentLoaded = 1.2 minutes.).  </p>

<p>I don’t think a decrease in performance this substantial is due to inefficient asp.net or SQL code changes inherent in this new release (No worries on me trying to cover for myself;  it wasn’t my code anyway ;-)).  In pursuit of this theory, I noticed that after the bulk of the Product page content would load in the browser, Chrome would then issue another get request (<a href="http://ourcompany.com/" rel="nofollow">http://ourcompany.com/</a> productpage.aspx?ImageControl_ctl00$ctl00$cphMainBodyFrame$ImageCompanyLogo=1) for the 12k company logo  which was housed in the master page asp.net ImageControl.  This request would take an average of an additional 40 seconds to complete.  So the logo wouldn’t show up until late in the game. </p>

<p>I ran Telerik’s JustTrace asp.net profiling tool my local DEV environment on the Product page which, despite the numerous calls to the database and data access layers, turned up nothing obvious in terms of performance bottlenecks.  Then I pasted the above logo url into the address while profiling in JustTrace and see that minimal processing is done to retrieve that logo.  Asp.net internally is calling only two stored procedures to the membership database to authenticate the user, along with some other minor system calls.  Despite the minimal asp.net and stored proc. calls, the request for the logo is taking far too long; and only for the Product page.  </p>

<p>Could this be a session lock issue?  Also, I’ve read thousands of SO posts over the years and this is my first post.  Thanks SO community for all your support over the years.  It’s been an invaluable resource.   </p>

<p>Win Server 2008 R2 (v6.1 Build 7601: SP1) - [VM with 8 CPU Cores > 32GB RAM]
IIS (7.5.7600.16385)
ASP.NET 4.0 WebForms 
MSSQL Server 12</p>