Microsoft Orleans grain communication performance
<p>I am working on a workflow engine using Mircosoft Orleans as the base, as it offers a number of useful features such as automatically distributing the work and handling fail over.</p>

<p>I have three types of grains:</p>

<ul>
<li>Workflow - Holds information in the workflow and what order work blocks should be executed in</li>
<li>Work Block - The parts that actually do the work</li>
<li>Execution - A single execution of the workflow</li>
</ul>

<p>My problem is that when running a large number of current executions, i.e. > 1000 the performance really suffers. I have done a bit of profiling and narrowed this down to the communication that happens between the grains. Is there anyway I can improve this any more?</p>

<p>Here is the outline of my code and how the grains interact</p>

<p>The execution grain sits in a loop, getting the next work block from the workflow and then calling execute on the workblock. It is this constant calling between grains that is causing the execution time for one of my test workflows to go from 10 seconds when running a single execution to around 5 minutes when running over 1000. Can this be improved or should I re-architect the solution to remove the grain communication?</p>

<pre><code>[StorageProvider(ProviderName = "WorkflowStore")]
[Reentrant]
[StatelessWorker]
public class Workflow : Grain&lt;WorkflowState&gt;, IWorkflow
{
    public Task&lt;BlockRef&gt; GetNext(Guid currentBlockId, string connectionName)
    {
         //Lookup the next work block
    }
}

[Reentrant]
[StatelessWorker]
public class WorkBlock : Grain&lt;WorkBlock State&gt;, IWorkBlock 
{
    public Task&lt;string&gt; Execute(IExecution execution)
    {
         //Do some work
    }
}


[StorageProvider(ProviderName = "ExecutionStore")]
public class Execution : Grain&lt;ExecutionState&gt;, IExecution, IRemindable
{
    private async Task ExecuteNext(bool skipBreakpointCheck = false)
    {            
        if (State.NextBlock == null)
        {
            await FindAndSetNext(null, null);
        }

        ...

        var outputConnection = await workblock.Execute();

        if (!string.IsNullOrEmpty(outputConnection))
        {
            await FindAndSetNext(State.NextBlock.Id, outputConnection);
            ExecuteNext().Ignore();
        }
    }

    private async Task FindAndSetNext(Guid? currentId, string outputConnection)
    {
        var next = currentId.HasValue ? await _flow.GetNextBlock(currentId.Value, outputConnection) : await _flow.GetNextBlock();
        ...
    }
}
</code></pre>