Cleaning up broken XML in Python
<p>A server I don't control sends broken XML with characters such as '>', '&amp;', '&lt;' etc. in attributes and text.</p>

<p>A small sample:</p>

<pre><code>&lt;StockFormula Description="" Name="F_Î”TURN" RankType="Higher" Scope="Universe" Weight="10.86%"&gt;
    &lt;Formula&gt;AstTurnTTM&gt;AstTurnPTM&lt;/Formula&gt;
&lt;/StockFormula&gt;
&lt;Composite Name="Piotroski &amp; Trends - &lt;11@4w600k 70b" Weight="0%" RankType="Higher"&gt;
&lt;/Composite&gt;
</code></pre>

<p>I settled on using the lxml module because it is case sensitive, is very fast and gets the job done. </p>

<p>How would I go about fixing up this type of XML? Basically, I am trying to replace all occurrences of the invalid characters with the proper escape sequences.</p>

<pre><code>import re

broken = '&lt;StockFormula Description="" Name="F_Î”TURN" RankType="Higher" Scope="Universe" Weight="10.86%"&gt;\n&lt;Formula&gt;AstTurnTTM&gt;AstTurnPTM&lt;/Formula&gt;\n&lt;Composite Name="Piotroski &amp; Trends - &lt;11@4w600k 70b" Weight="0%" RankType="Higher"&gt;\n&lt;/Composite&gt;'
print re.sub(r'(.*Name=".*)&amp;(")', r'\g&lt;1&gt;&amp;gt;\g&lt;2&gt;', broken)
</code></pre>

<p>Output:</p>

<pre><code>&lt;StockFormula Description="" Name="F_ÃŽâ€TURN" RankType="Higher" Scope="Universe" Weight="10.86%"&gt;
    &lt;Formula&gt;AstTurnTTM&gt;AstTurnPTM&lt;/Formula&gt;
&lt;/StockFormula&gt;
&lt;Composite Name="Piotroski &amp; Trends - &lt;11@4w600k 70b" Weight="0%" RankType="Higher"&gt;
&lt;/Composite&gt;
</code></pre>