Download file from a dynamically generated link which lies in the source code of an HTML
<p>I am trying to get the Weather data from BOM Australia. The manual way is to go to <a href="http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064" rel="nofollow">http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064</a> and click 'All years of data', and it downloads the file!</p>

<p>Here's what I have tried to automate this:</p>

<pre><code>using (WebClient client = new WebClient())
            {

                string html = client.DownloadString("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064");


                List&lt;string&gt; list = LinkExtractor.Extract(html);
                foreach (var link in list)
                {
                    if (link.StartsWith("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile"))
                    {

                        string resource = "http://www.bom.gov.au" + link;
                        MessageBox.Show(resource);


                        client.DownloadFileAsync(new Uri(resource), Dts.Connections["data.zip"].ConnectionString);
                        break;
                    }
                }




            }
</code></pre>

<p>Don't worry about the linkExtractor, it works as I am able to see the link that gives the file. The problem is that the 'DownloadFileAsync' creates a new request which does not let the file to get downloaded since the file needs the same session.</p>

<p>Is there a way I can do this? Please reach out for more clarification.</p>

<p>UPDATE:</p>

<p>Here are the changes I made, utilising cookies from HttpWebRequest. However, I am still not able to download the file.</p>

<pre><code>HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064");
            request.CookieContainer = new CookieContainer();

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

            foreach (Cookie cook in response.Cookies)
            {
                MessageBox.Show(cook.ToString());
            }

            if (response.StatusCode == HttpStatusCode.OK)
           {
                Stream receiveStream = response.GetResponseStream();
                StreamReader readStream = null;

                if (response.CharacterSet == null)
                {
                    readStream = new StreamReader(receiveStream);
                }
                else
                {
                    readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
                }

                string data = readStream.ReadToEnd();



                using (WebClient client = new WebClient())
                {
                    foreach (Cookie cook in response.Cookies)
                    {
                        MessageBox.Show(cook.ToString());
                        client.Headers.Add(HttpRequestHeader.Cookie, cook.ToString());
                    }

                    List&lt;string&gt; list = LinkExtractor.Extract(data);
                    foreach (var link in list)
                    {
                        if (link.StartsWith("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile"))
                        {

                            string initial = "http://www.bom.gov.au" + link;
                            MessageBox.Show(initial);

                            //client.Headers.Add(HttpRequestHeader.Cookie, "JSESSIONID=2EBAFF7EFE2EEFE8140118CE5170B8F6");
                            client.DownloadFile(new Uri(initial), Dts.Connections["data.zip"].ConnectionString);
                            break;
                        }
                    }




                }

                response.Close();
                readStream.Close();
            }
</code></pre>