Zend framework 2 : Add different authentication ad

I have two different modules. Now I need to add different authentication mechanism for both modules.

So I added event code first module's Module.php's onBootstrap method

$listener = $serviceManager->get('First\Service\AuthListener');
$listener->setAdapter($serviceManager->get('First\Service\BasicAuthAdapter'));
$eventManager->attach(MvcEvent::EVENT_ROUTE, $listener, 0);

and in second module's Module.php's onBootstrap method

$listener = $serviceManager->get('Second\Service\AuthListener');
$listener->setAdapter($serviceManager->get('Second\Service\AdvAuthAdapter'));
$eventManager->attach(MvcEvent::EVENT_ROUTE, $listener, 0);

Now if I disable one of modules, functionality working fine and request properly authenticated. While enabling both module do some kind of overlapping So even required module properly authenticated, But other module event code also got executed and system give not authenticated error.

I am thinking this due to event handler code in both module.php is executed without take care of requested module url.

I can verify with requested route pattern before authentication, But that is look like a hack instead of good solution.

If better solution exists for handling this issue ?

UPDATE : My AuthListener Code :

namespace First\Service;
use Zend\Authentication\Adapter\AdapterInterface;
use Zend\Mvc\MvcEvent;

class AuthListener
{
    protected $adapter;

    public function setAdapter(AdapterInterface $adapter)
    {
        $this->adapter = $adapter;
    }

    public function __invoke(MvcEvent $event)
    {
        $result = $this->adapter->authenticate();

        if (!$result->isValid()) {

            $response = $event->getResponse();

            // Set some response content
            $response->setStatusCode(401);

            $routeMatch = $event->getRouteMatch();
            $routeMatch->setParam('controller', 'First\Controller\Error');
            $routeMatch->setParam('action', 'Auth');
        }
    }
}

There is a good way to make module specific bootstrap - to use SharedManager:

$e->getApplication()->getEventManager()->getSharedManager()
  ->attach(__NAMESPACE__, 'dispatch', function(MvcEvent $e) {

     // This code will be executed for all controllers in current __NAMESPACE__

}, 100);

Here is a good article to understand difference between EventManager and SharedEventManager

There is no additional info about listeners in the question, but I try to guess:

If you use as listener some callable class - it's ok, just replace function() { } by your $listener.
If you use as listener some class, that implements ListenerAggregateInterface, you should convert listeners to SharedListenerAggregateInterface and use method attachAggregate instead of attach

I hope it helps!