clustered consistent hash pool producing new route

Question

I have a solution with 2 command line projects that creates an akka.net cluster with a seed and client process. The seed starts the cluster and then instantiates a consistent-hash-cluster-router that performs hash mapping on any message that implements my interface "IHasRouting". So any IHasRouting message (from the seed or client) should end up at seed on a routee for the hash of that message.

The projects start fine and the cluster forms without error. Both seed and client instantiate a router. All messages from the seed and client have the same "VolumeId" so they should go to the same route at the seed. BUT Messages from the client node result in a new routee for those messages at the seed!

My understanding of a consistent-hash-cluster-router is that:

• An IActorRef represting it should exit on each node where actors in that node intend to send messages to the router.
• The implentation of the router should be identical on each node and have the same actor name.
• All messages to the router should implement IConsistentHash or the router instance should have a "WithHashMapping()"
• All messages with the same hash will arrive at only one routee and it will always be the same routee
• A routee may take more than one hash

I believe I understand how a consistent-hash-cluster-router should behave and plenty of the DEVs would appear to be using the router type correctly so my implementation must be wrong... Please help! I can provide the full solution if that helps.

The code that creates the router:

system.ActorOf(
   new ClusterRouterPool(
       local: new ConsistentHashingPool(nrOfInstances: 1)
          .WithHashMapping(m => (m as IHasRouting)?.Company?.VolumeId ?? throw new Exception("no routing!")),
                settings: new ClusterRouterPoolSettings(
                    100,
                    100,
                    allowLocalRoutees: allowLocalRoutees, //true if the node role is a Seed
                    useRole: "Seed"))
                    .Props(Props.Create(() => new CompanyDeliveryActor())), "company-router");

I have a "Company" class that is required for messages to the router. All VolumeIds are the same for this test.

public class Company
{
    public readonly Guid CompanyId;
    public readonly Guid VolumeId;
    public readonly string CompanyName;
    public Company(Guid companyId, Guid volumeId, string companyName)
    {
        this.CompanyId = companyId;
        this.VolumeId = volumeId;
        this.CompanyName = companyName;
    }
}

The IHasRouting interface that is used by the router mapping:

public interface IHasRouting
{
    Company Company { get; }
}

An example message class that can be sent to the router:

public class GetTripsMessage : IHasRouting
{
    public Company Company { get; private set; }
    public GetTripsMessage(Company company)
    {
        this.Company = company;
    }
}

And Finally the CompanyDeliverActor which is instantiated for each routee at the router:

public class CompanyDeliveryActor : ReceiveActor
{
    private readonly Dictionary<Guid, IActorRef> companyManagers = new Dictionary<Guid, IActorRef>();
    private readonly Guid instanceid = Guid.NewGuid();

    public CompanyDeliveryActor()
    {
        this.Receive<GetTripsMessage>(m => this.RouteCompanyMessage(m, m.Company));
        this.Receive<SetTripsMessage>(m => this.RouteCompanyMessage(m, m.Company));
    }

    private void RouteCompanyMessage(object m, Company company)
    {
        //placing a watch here shows that this.instanceid is different for messages from the client.

        if (!this.companyManagers.TryGetValue(company.CompanyId, out var manager))
        {
            manager = Context.ActorOf(Props.Create(() => new CompanyManagerActor()));
            this.companyManagers[company.CompanyId] = manager;                
        }

        manager.Tell(m, Context.Sender);
    }
}

Thanks for any guidance.

Answer 1

After investigation I agree that my understanding of the clustered-consistent-hash-router is wrong. I now understand that I need to communicate the IActorRef that represents the router to the rest of the cluster. This might be accomplished via a ClusterSinglton or (possibly) a distributed pub sub mechanism of some type. A mechanism needs to handle the situation where the node or Actor becomes "lost", a new Actor (with new routees) is instantiated and the new Actor is communicated to all those in the cluster that had the previous reference.

I am investigating the Cluster Sharding approach and will post another question in regards to technical difficulties that I am having with that approach.

Answer 2

When you're calling ActorOf you're actually creating a new actor instance on the current cluster node. In case of pool routers, creating new router will also create a new pool of routee actors, which may be dispatched over other nodes. This also means, that is you're calling ActorOf two times (one on client node, one on seed node) in result you'll receive two separate pools of actors.

1. The easiest way to solve this is to simply have router pool instantiated only once - let it dispatch pool of routees across cluster nodes - and forward all of your requests through node, which serves as an entry point to your cluster. You can configure other nodes as failover to retake that task in entry node fails.
2. Another thing to do is to use either Distributed Pub/Sub or cluster sharding depending on what are you using cluster message routing for.