unicode string comparison in C
<p>I'm learning UNIX system programming.  I'm writing a simple shell application for UNIX (I'm on OS X Yosemite ver 10.10.5 and I use Xcode). I had some experience with C but not much.</p>

<p>Utility programs work fine and will print unicode characters (though ls prints '????' instead of it in Xcode console, but it seems to be the problem of the debugger itself).</p>

<p>I've made a little research and found out that <code>strcmp()</code> should work fine with it too, as far as it just compares bytes and looks for a zero byte in the end. Reading input should be ok too, as you just read bytes.</p>

<p>I've also read that unicode string shouldn't contain null bytes. However, some input will cause <code>EXC_BAD_ACCESS</code> when doing `strcmp().</p>

<p>Code:</p>

<p>Reading user input:</p>

<pre><code>char* readCommand(void) {
    int buffer_size = LINE_BUFFER_SIZE;
    char *buffer = malloc(sizeof(char) * buffer_size);
    int position = 0;
    int character;

    if(!buffer)
    {
        fprintf(stderr, "readCommand failed: memory allocation error");
        exit(ALLOCATION_ERROR);
    }

    while (1) {
        character = getchar();
        if(character == EOF || character == '\n')
        {
            buffer[position] = '\0';
            char* cmd = buffer;
            free(buffer);
            return cmd;
        }
        else {
            buffer[position] = character;
        }
        if(++position &gt;= sizeof(buffer))
        {
            buffer_size += LINE_BUFFER_SIZE;
            buffer = realloc(buffer, sizeof(char) * buffer_size);
            if(!buffer) {
                fprintf(stderr, "readCommand failed: memory reallocation error");
                free(buffer);
                exit(ALLOCATION_ERROR);
            }
        }
    }
    return NULL;
}
</code></pre>

<p>Split args:</p>

<pre><code>int split_string_quotes(char* source, char** argv, size_t arg_count)
{
    enum split_states state = DULL;
    char* p, *word_start = NULL;
    int character;
    int argc = 0;
    for(p = source; argc &lt; arg_count &amp;&amp; *p != '\0'; p++)
    {
        character = (unsigned char) *p;
        switch (state) {
            case DULL:
                if(isspace(character))
                {
                    continue;
                }
                if(character == '"')
                {
                    state = IN_STRING;
                    word_start = p+1;
                    continue;
                }
                state = IN_WORD;
                word_start = p;
                continue;

            case IN_WORD:
                if(isspace(character))
                {
                    state = DULL;
                    *p = 0;
                    argv[argc++] = word_start;
                }
                continue;

            case IN_STRING:
                if(character == '"')
                {
                    state = DULL;
                    *p = 0;
                    argv[argc++] = word_start;
                }
                continue;
        }
    }

    if(state != DULL &amp;&amp; argc &lt; arg_count)
    {
        argv[argc++] = word_start;
    }
    argv[argc] = NULL;
    return argc;
}
</code></pre>

<p>That's where strcmp is:</p>

<pre><code>int shell_execute(char **args)
{
    for(int i = 0; i &lt; 3; i++)
    {
        if(strcmp(args[0], commands[i]) == 0)
        {
            return (*standardFuncs[i])(args);
        }
    }
    shell_launch(args);
    return 0;
}
</code></pre>

<p>And the main loop</p>

<pre><code>    char* current_dir = malloc(sizeof(char)*PATH_MAX);
    char* args[MAX_ARGS];
    char* command;
    printf("dolphinShell (c) Alex Kale 2016\n");
    while (1)
    {
        getwd(current_dir);
        printf("dsh: %s-&gt; ", current_dir);
        command = readCommand();
        printf("%s\n", command);
        split_string_quotes(command, args, MAX_ARGS);
        if(shell_execute(args) == -1) break;
    }
    free(current_dir);
    return 0;
</code></pre>

<p>So, the problem is that some unicode strings I type work fine and never cause <code>EXC_BAD_ACCESS</code>, but when I type <code>фывпфвыапы</code>, for example, it breaks. I think the problem is with accessing <code>args[0]</code>, but here's the debugger's output:</p>

<pre><code> Printing description of args:
    (char **) args = 0x00007fff5fbff900
    *args   char *  0x101800a00 0x0000000101800a00
    Printing description of *(*(args)):
    (char) **args = '\xd1'
</code></pre>

<p>So it thinks that <code>args[0]</code> is empty, but is it empty? Or is it confused by all the zeroes?</p>

<p>I'm really confused, I've spent a lot of time researching and seem to be stuck here.</p>

<p>I have also tried using <code>wchar_t</code> and <code>wcscmp()</code>, but it doesn't work good with <code>execvp()</code> and doesn't solve the problem.</p>

<p>I have also tried <code>gcc -Wall -Wextra</code> and here's the output:</p>

<pre><code>main.c:53:26: warning: comparison of integers of different signs: 'int' and
      'size_t' (aka 'unsigned long') [-Wsign-compare]
    for(p = source; argc &lt; arg_count &amp;&amp; *p != '\0'; p++)
                    ~~~~ ^ ~~~~~~~~~
main.c:92:30: warning: comparison of integers of different signs: 'int' and
      'size_t' (aka 'unsigned long') [-Wsign-compare]
    if(state != DULL &amp;&amp; argc &lt; arg_count)
                        ~~~~ ^ ~~~~~~~~~
main.c:124:23: warning: comparison of integers of different signs: 'int' and
      'unsigned long' [-Wsign-compare]
        if(++position &gt;= sizeof(buffer))
           ~~~~~~~~~~ ^  ~~~~~~~~~~~~~~
main.c:180:18: warning: unused parameter 'args' [-Wunused-parameter]
int dHelp(char **args)
                 ^
main.c:203:18: warning: unused parameter 'args' [-Wunused-parameter]
int dExit(char **args)
                 ^
main.c:210:14: warning: unused parameter 'argc' [-Wunused-parameter]
int main(int argc, const char** argv)
             ^
main.c:210:33: warning: unused parameter 'argv' [-Wunused-parameter]
int main(int argc, const char** argv)
                                ^
7 warnings generated.
</code></pre>

<p>But I don't think that's the case (correct me if I'm wrong).</p>