Django query Unicode Issues
<h2><strong>EDIT #2:</strong></h2>
<pre><code>{'sql': 'SELECT &quot;strains_terpene&quot;.&quot;id&quot;, &quot;strains_terpene&quot;.&quot;name&quot;, 
&quot;strains_terpene&quot;.&quot;short_desc&quot;, &quot;strains_terpene&quot;.&quot;long_desc&quot;, 
&quot;strains_terpene&quot;.&quot;aroma&quot;, &quot;strains_terpene&quot;.&quot;flavor&quot;, 
&quot;strains_terpene&quot;.&quot;effects&quot; FROM &quot;strains_terpene&quot; WHERE     
&quot;strains_terpene&quot;.&quot;name&quot; = \'\xce±-Humulene\'', 'time': '0.000'}
</code></pre>
<p>Upon closer look it appears that django may be properly escaping the single quotes in the end. Had to take a different angle to see this by using this:</p>
<pre><code>from django.db import connections
connections['default'].queries
</code></pre>
<p><em><strong>So now the question remains, why even though python3, django, and postgres are all set to utf-8 is the unicode being encoded to local in the query?</strong></em></p>
<h1><strong>Original Question:</strong></h1>
<p>Here is the runtime error:</p>
<pre><code>strains.models.DoesNotExist: Terpene matching query does not exist.
</code></pre>
<p>Here is the <code>str(Terpene.objects.filter(name='β-Caryophyllene').query)</code>:</p>
<pre><code>SELECT &quot;strains_terpene&quot;.&quot;id&quot;, &quot;strains_terpene&quot;.&quot;name&quot;, &quot;strains_terpene&quot;.&quot;short_desc&quot;, &quot;strains_terpene&quot;.&quot;long_desc&quot;, &quot;strains_terpene&quot;.&quot;aroma&quot;, &quot;strains_terpene&quot;.&quot;flavor&quot;, &quot;strains_terpene&quot;.&quot;effects&quot; 
FROM &quot;strains_terpene&quot; 
WHERE &quot;strains_terpene&quot;.&quot;name&quot; = ß-Caryophyllene
</code></pre>
<p>Here is how postgres likes to see the query for it to work:</p>
<pre><code>select * from strains_terpene where name = 'β-Caryophyllene'
</code></pre>
<p>Am i missing something here? Why is Django not wrapping my condition in single quotes?</p>
<ul>
<li>PostgresDB is encoded with utf-8</li>
<li>Python 3 strings are unicode</li>
</ul>
<h2><strong>EDIT:</strong></h2>
<p>I notice the query attribute is also converting the β to ß...</p>
<p>I thought this could be a conversion issue considering im using windows cmd for the python shell.</p>
<p>So i did a:</p>
<pre><code>with open('log2.txt','w',encoding='utf-8') as f:
    print(Terpene.objects.filter(name='β-Caryophyllene').query, file=f)
</code></pre>
<p>And here are the results even when output directly to utf-8 plain text.</p>
<pre><code>SELECT &quot;strains_terpene&quot;.&quot;id&quot;, &quot;strains_terpene&quot;.&quot;name&quot;, &quot;strains_terpene&quot;.&quot;short_desc&quot;, &quot;strains_terpene&quot;.&quot;long_desc&quot;, &quot;strains_terpene&quot;.&quot;aroma&quot;, &quot;strains_terpene&quot;.&quot;flavor&quot;, &quot;strains_terpene&quot;.&quot;effects&quot; 
FROM &quot;strains_terpene&quot; 
WHERE &quot;strains_terpene&quot;.&quot;name&quot; = ß-Caryophyllene
</code></pre>
<p>So now I am confused on 2 fronts. Why does django choose to ommit the single quotes for the where condition and why is the lowercase beta being converted to an uppercase?</p>
<h3><strong>EXTRA INFO:</strong></h3>
<p>Here is the section of actual code.</p>
<ul>
<li>Importing mass results via CSV.</li>
<li>The results dict stores the mapping between columns and Terpene Names</li>
<li>The first log.txt is for verifying the contents of results</li>
<li>The second log1.txt is to verify the key before using it as the lookup condition</li>
<li>The finally log2.txt verifies sql being sent to the database</li>
</ul>
<p>First the Code Snippet:</p>
<pre><code>results = {
            u'α-Pinene': row[7],
            u'β-Pinene': row[8],
            u'Terpinolene': row[9],
            u'Geraniol': row[10],
            u'α-Terpinene': row[11],
            u'γ-Terpinene': row[12],
            u'Camphene': row[13],
            u'Linalool': row[14],
            u'd-Limonene': row[15],
            u'Citral': row[16],
            u'Myrcene': row[17],
            u'α-Terpineol': row[18],
            u'Citronellol': row[19],
            u'dl-Menthol': row[20],
            u'1-Borneol': row[21],
            u'2-Piperidone': row[22],
            u'β-Caryophyllene': row[23],
            u'α-Humulene': row[24],
            u'Caryophyllene Oxide': row[25],
        }
        with open(&quot;log.txt&quot;, &quot;w&quot;) as text_file:
            print(results.keys(), file=text_file)
        for r, v in results.items():
            if '&lt;' not in v:
                value = float(v.replace(&quot;%&quot;, &quot;&quot;))
                with open(&quot;log1.txt&quot;, &quot;w&quot;) as text2:
                    print(r, file=text2)
                with open(&quot;log2.txt&quot;, &quot;w&quot;, encoding=&quot;utf-8&quot;) as text3:
                    print(Terpene.objects.filter(name=r).query, file=text3)
                TerpeneResult.objects.create(
                    terpene=Terpene.objects.get(name=r),
                    qa_sample=sample,
                    result=value,
                )
</code></pre>
<p><strong>And log.txt</strong>  -- results.keys():</p>
<pre><code>dict_keys(['dl-Menthol', 'Geraniol', 'Camphene', '1-Borneol', 'Linalool', 
'α-Humulene', 'Caryophyllene Oxide', 'β-Caryophyllene', 'Citronellol', 
'α-Pinene', '2-Piperidone', 'β-Pinene', 'd-Limonene', 'γ-Terpinene',  
'Terpinolene', 'α-Terpineol', 'Myrcene', 'α-Terpinene', 'Citral'])
</code></pre>
<p><strong>log1.txt</strong> -- <code>α-Humulene</code></p>
<p>Lastly the sql being generated -- <strong>log2.txt</strong>:</p>
<pre><code>SELECT &quot;strains_terpene&quot;.&quot;id&quot;, &quot;strains_terpene&quot;.&quot;name&quot;, &quot;strains_terpene&quot;.&quot;short_desc&quot;, &quot;strains_terpene&quot;.&quot;long_desc&quot;, &quot;strains_terpene&quot;.&quot;aroma&quot;, &quot;strains_terpene&quot;.&quot;flavor&quot;, &quot;strains_terpene&quot;.&quot;effects&quot; 
FROM &quot;strains_terpene&quot; 
WHERE &quot;strains_terpene&quot;.&quot;name&quot; = Î±-Humulene
</code></pre>
<p>Note the unicode being lost at the last moment when the sql is generated.</p>