MFC application crashing in ProcessShellCommand() when file to open specified on command line
<p>The problem I need to solve is how to use the MFC function <code>ProcessShellCommand()</code> in the <code>InitInstance()</code> of a <code>CWinApp</code> to process a File Open with a specific path when the application to open the file is being launched by another application.</p>

<p>I have an MFC MDI (Multiple Document Interface) application that is started by another application with a command line using <code>ShellExecute()</code> containing the path to a file to open. When compiled with Visual Studio 2005, I do not see a problem with the launched application. When compiled with Visual Studio 2013, the application that is started crashes and I never see the application window.</p>

<p>Running in the debugger, I see an error dialog that has a title of "Microsoft Visual C++ Runtime Library" with an error message of "Debug Assertion Failed!" specifing the mfc120ud.dll and file of src\mfc\filelist.cpp line: 221</p>

<p>At this point I can attach to the application process then click the Retry button of the dialog. Then when I continue I see a Visual Studio error dialog from an unhandled exception that seems to be generated by <code>KernelBase.dll</code>.</p>

<blockquote>
  <p>Unhandled exception at 0x76EBC54F in NHPOSLM.exe: Microsoft C++
  exception: CInvalidArgException at memory location 0x0014F094.</p>
</blockquote>

<p>If I click the Continue button, I get another "Debug Assertion Failed" this time from src\mfc\filelist.cpp line: 234</p>

<p>After making a source change to perform a <code>Sleep()</code> in order to use the <code>Debug-&gt;Attach to process</code> Visual Studio 2013 command I was able to use the debugger to look at various data areas and step through code.</p>

<p>At one point, after stepping over the <code>ProcessShellCommand()</code> function and seeing the exception, when the thread returned to the statement after the function call, I used the set source line debugger command to set the current line back to the function call and stepped over it again. This time there was no exception and when I allowed the thread to continue, the application came up with the correct file open.</p>

<p>Then I found this article, <a href="http://blog.rectorsquid.com/processshellcommand-and-the-view-and-frame-windows/" rel="nofollow">ProcessShellCommand and the View and Frame Windows</a> which states the following:</p>

<blockquote>
  <p>The problem is that the code in ProcessShellCommand() opens the
  document file before it finishes creating the frame and view windows.
  Those windows exist but there is no way to access them because the
  frame window pointer is not saved to an app-wide variable until after
  the document is open.</p>
</blockquote>

<p>The solution provided in the article is to call <code>ProcesShellCommand()</code> twice as in the following code segment.</p>

<pre><code>CCommandLineInfo cmdInfo;

if( !ProcessShellCommand( cmdInfo ) )
    return FALSE;

ParseCommandLine( cmdInfo );

if( cmdInfo.m_nShellCommand != CCommandLineInfo::FileNew )
{
    if (!ProcessShellCommand( cmdInfo ) )
        return FALSE;
}
</code></pre>

<p>I have tried this approach in my application and it does indeed open the document and seems to process everything correctly. The problem is that while this works for an SDI (Single Document Interface) type of MFC application for an MDI (Multiple Document Interface) type of MFC application you will see two document windows, an empty one as created by File New and the one actually wanted created by File Open.</p>

<p>I have also found that using the debugger to attach to the application process and then stepping through slowly, if I let the launched application Continue after the exception dialogs, the application will finish coming up with the requested file. However if not in the debugger, the launched application's main window will not display.</p>

<p>So there appears to be some kind of a race condition for the environment to be ready for the launched application to have its run time environment fully initialized.</p>

<p>For an explanation of the <code>ProcessShellCommand()</code> function see <a href="https://msdn.microsoft.com/en-us/library/zatkyh2a.aspx" rel="nofollow">CWinApp::ProcessShellCommand</a> which describes the basic process for command line processing as:</p>

<ol>
<li>After being created in <code>InitInstance</code>, the <code>CCommandLineInfo</code> object is
passed to <code>ParseCommandLine</code>.</li>
<li><code>ParseCommandLine</code> then calls <code>CCommandLineInfo::ParseParam</code> repeatedly,
once for each parameter.</li>
<li><code>ParseParam</code> fills the <code>CCommandLineInfo</code> object, which is then passed
to <code>ProcessShellCommand</code>.</li>
<li><code>ProcessShellCommand</code> handles the command-line arguments and flags.</li>
</ol>

<p>The specific source we are using in the <code>InitInstance()</code> is:</p>

<pre><code>// Register the application's document templates.  Document templates
//  serve as the connection between documents, frame windows and views.

CMultiDocTemplate* pDocTemplate;
pDocTemplate = new CMultiDocTemplate(
    IDR_NEWLAYTYPE,
    RUNTIME_CLASS(CNewLayoutDoc),
    RUNTIME_CLASS(CChildFrame), // custom MDI child frame
    RUNTIME_CLASS(CNewLayoutView/*CLeftView*/));
AddDocTemplate(pDocTemplate);

// create main MDI Frame window
CMainFrame* pMainFrame = new CMainFrame;
if (!pMainFrame-&gt;LoadFrame(IDR_MAINFRAME))
    return FALSE;
m_pMainWnd = pMainFrame;

// Parse command line for standard shell commands, DDE, file open
CLOMCommandLineInfo cmdInfo;
/*initialize language identifier to English so we wont have garbage if no language 
flag is set on teh command line*/
cmdInfo.lang = LANG_ENGLISH;
cmdInfo.sublang = SUBLANG_ENGLISH_US;
//CCommandLineInfo cmdInfo;
ParseCommandLine(cmdInfo);

BOOL success = pMainFrame-&gt;ProcessCmdLineLang(cmdInfo.lang, cmdInfo.sublang);
if(!success){
    AfxMessageBox(IDS_CMDLINE_LANG_NF,MB_OK,0);
}
// Dispatch commands specified on the command line
if (!ProcessShellCommand(cmdInfo))
    return FALSE;

// The main window has been initialized, so show and update it.
pMainFrame-&gt;ShowWindow(SW_SHOWNORMAL);
pMainFrame-&gt;UpdateWindow();
</code></pre>

<p>I do not like the solution provided in the article of calling <code>ProcessShellCommand()</code> twice as it appears untidy. It does not provide what I need for an MDI application. I do not know why this code seems to work fine with VS 2005 and cause an error in VS2013.</p>

<p>Finally I came across this posting in codeproject, <a href="http://www.codeproject.com/Questions/231741/Debug-Assertion-Error-Visual-Studio" rel="nofollow">Debug Assertion Error Visual Studio 2010</a>, which indicated that a similar assertion error involving src\mfc\filelist.cpp was traced to adding a file path to the Recent File List when the file path contained an asterisk.</p>

<p>When I use the debugger to look at the <code>cmdInfo</code> object there is a member, <code>(*((CCommandLineInfo*)(&amp;(cmdInfo)))).m_strFileName</code>, which contains a value of L"C:\Users\rchamber\Documents\ailan_221.dat". This is the correct path from the command line provided by the application that is starting the launched application with <code>ShellExecute()</code>.</p>

<p><strong>Note:</strong> each of the back slashes in the string are actually double backslashes in the debug watch. So to render in stack overflow properly I need to add additional backslashes as in L"C:\\Users\\rchamber\\Documents\\ailan_221.dat" however the double backslash seems to be what the debugger uses to represent a single backslash character.</p>

<p><strong>Edit 3/23/2016 - note concerning source history</strong></p>

<p>One additional bit of information is the source history for this application. The original application was created using Visual Studio 6.0 then moved to Visual Studio 2005. The <code>InitInstance()</code> method of the <code>CWinApp</code> has not been modified to any degree since it was originally created.</p>