Trouble with strings with <U+0092> Unicode characters
<p>I have a very large dataset (70k rows, 2600 columns, CSV format) that I have created by web scraping. Unfortunately, doing the pre-processing, processing etc. at some point some problematic characters have become encoded in an odd way and I have problems dealing with them.</p>

<p>I have strings like the following:</p>

<pre><code>x = "but it doesn&lt;U+0092&gt;t matter"
</code></pre>

<p><a href="http://www.fileformat.info/info/unicode/char/92/index.htm" rel="noreferrer">Looking up the code</a>, we can see that it should be the character <code>’</code>, which actually should be <code>'</code> (the data are user-generated so may contain all kinds of odd characters). Although from looking that character, it seems that others also have problems with it (<a href="https://stackoverflow.com/questions/8571648/how-do-i-get-rid-of-this-unicode-character">1</a>, <a href="https://stackoverflow.com/questions/11030851/jquery-parsejson-u0092-character-is-not-parsed">2</a>, <a href="https://github.com/tenderlove/psych/issues/271" rel="noreferrer">3</a>). It's labelled a control character, not sure what that is, but perhaps that's why it's so hard to deal with.</p>

<p>Most of the other questions about Unicode in R concern Unicode in the format like this <code>\u0092</code>.</p>

<h2>Just use <code>Encoding()</code></h2>

<p>Let's try:</p>

<pre><code>#&gt; x = "but it doesn&lt;U+0092&gt;t matter"
#&gt; Encoding(x)
#[1] "unknown"
#&gt; Encoding(x) = "UTF-8"
#&gt; Encoding(x)
#[1] "unknown"
#&gt; x
#[1] "but it doesn&lt;U+0092&gt;t matter"
</code></pre>

<p>So this does not seem to do anything.</p>

<h2>Use the hack functions from these previous questions</h2>

<p>There are a few prior questions that concern this Unicode format and try to convert them:</p>

<ul>
<li><a href="https://stackoverflow.com/questions/33866570/display-unicode-in-r">Display unicode in R</a></li>
<li><a href="https://stackoverflow.com/questions/28248457/gsub-in-r-with-unicode-replacement-give-different-results-under-windows-compared?rq=1">gsub in R with unicode replacement give different results under Windows compared with Unix?</a></li>
</ul>

<p>Oddly, the example they give work, but mine doesn't.</p>

<pre><code>#&gt; test.string &lt;- "This is a &lt;U+03B1&gt; &lt;U+03B2&gt; &lt;U+03B2&gt; &lt;U+03B3&gt; test &lt;U+03B4&gt; string."
#&gt; Encoding(test.string)
#[1] "unknown"
#&gt; to_true_unicode(test.string)
#[1] "This is a α β β γ test δ string."
</code></pre>

<p>But:</p>

<pre><code>#&gt; x2 = to_true_unicode(x)
#&gt; x2
#[1] "but it doesn\u0092t matter"
#&gt; cat(x2)
#but it doesnt matter
#&gt; Encoding(x2)
#[1] "UTF-8"
</code></pre>

<p>So, it managed to convert to the <code>\u</code> format from the &lt;<code>U+....&gt;</code> format, and using <code>cat()</code> prints the character without that symbol (or a bugged symbol on SO).</p>

<h2>Just search and replace them manually</h2>

<p>I only have a limited number of these problems, so I could perhaps just use search-replace to solve it. However:</p>

<pre><code>#&gt; #base-r
#&gt; gsub(x = x, pattern = "&lt;U+0092&gt;", replacement = "'")
#[1] "but it doesn&lt;U+0092&gt;t matter"
#&gt; #stringr/stringi
#&gt; library(stringr)
#&gt; str_replace(x, pattern = "&lt;U+0092&gt;", "'")
#[1] "but it doesn&lt;U+0092&gt;t matter"
</code></pre>

<p>So replacement does not seem to work, but it does work on the <code>\u</code> versions:</p>

<pre><code>#&gt; #base-r
#&gt; gsub(x = x2, pattern = "\u0092", replacement = "'")
#[1] "but it doesn't matter"
#&gt; #stringr/stringi
#&gt; library(stringr)
#&gt; str_replace(x2, pattern = "\u0092", "'")
#[1] "but it doesn't matter"
</code></pre>

<p>So, this suggests a working method: 1) convert <code>&lt;U+&gt;</code> format to <code>\u</code> format, then use search-replace.</p>

<h2>Unescape with <code>stringi::stri_unescape_unicode()</code></h2>

<p>Does not seem to work with either version:</p>

<pre><code>#&gt; stringi::stri_unescape_unicode(x)
#[1] "but it doesn&lt;U+0092&gt;t matter"
#&gt; stringi::stri_unescape_unicode(x2)
#[1] "but it doesn\u0092t matter"
</code></pre>

<p>Is there some generally applicable way to deal with problems like this?</p>

<h2>My setup</h2>

<p>My sessionInfo is:</p>

<pre><code>&gt; sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows &gt;= 8 x64 (build 9200)

locale:
[1] LC_COLLATE=Danish_Denmark.1252  LC_CTYPE=Danish_Denmark.1252    LC_MONETARY=Danish_Denmark.1252
[4] LC_NUMERIC=C                    LC_TIME=Danish_Denmark.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] stringr_1.0.0

loaded via a namespace (and not attached):
[1] magrittr_1.5  tools_3.2.3   stringi_1.0-1
</code></pre>

<p>Running R via RStudio (0.99.893, preview) on Windows 8.1, 64-bit. Keyboard and time-units are Danish, but everything else is in English.</p>