How to decode a webpage into UTF8 regardless of its encoding
<p>I'm using <code>WebClient.DownloadFile</code> in C# to download web pages. They download fine, but the encoding is often ambiguous (no BOM present). The Mozilla Universal Charset Detector (<a href="https://github.com/superstrom/chardetsharp" rel="nofollow">port 1</a>, <a href="https://github.com/errepi/ude" rel="nofollow">port 2</a>) provides a system to detect the encoding of plaintext files using heuristics and <a href="http://www.architectshack.com/TextFileEncodingDetector.ashx" rel="nofollow">this</a> provides an even simpler method that recognizes fewer encodings.</p>

<p>So firstly, are HTML web pages even encoded in strange encodings like Shift-JIS and EUC-KR? If not, then a faster detection method can be used that assumes something is ASCII/ANSI or UTF8. </p>

<p>Secondly, even after detecting the encoding, how does one go about decoding the <code>byte[]</code> of the file into an appropriate UTF8 string? And after doing some string processing, can I save the file back to disk with a UTF8 BOM? Or would I have to add extra tags like <code>&lt;meta charset="utf-8".."&gt;</code> into the HTML file as well?</p>