Choose adapter dynamically depending on librarie(s

Question

I am designing a library that has adapters that supports a wide-range of libraries. I want the library to dynamically choose which ever adapter that has the library it uses installed on the machine when importing specific classes.

The goal is to be able to change the library that the program depends on without having to make modifications to the code. This particular feature is for handling RabbitMQ connections, as we have had a lot of problems with pika, we want to be able to change to a different library e.g. pyAMPQ or rabbitpy without having to change the underlying code.

I was thinking of implementing something like this in the __init__.py file of servicelibrary.simple.

try:
    #import pika # Is pika installed?
    from servicelibrary.simple.synchronous import Publisher
    from servicelibrary.simple.synchronous import Consumer
except ImportError:
    #import ampq # Is ampq installed?
    from servicelibrary.simple.alternative import Publisher
    from servicelibrary.simple.alternative import Consumer

Then when the user imports the library

from servicelibrary.simple import Publisher

The underlying layer looks something like this

alternative.py

import amqp

class Publisher(object):
    ......

class Consumer(object):
     ......    

synchronous.py

import pika

class Publisher(object):
    ......

class Consumer(object):
     ......   

This would automatically pick the second one when the first one is not installed.

Is there a better way of implementing something like this? If anyone could link a library/adapter with a similar implementation that would be helpful as well.

[Edit]

What would be the cleanest way to implement something like this? In the future I would also like to be able to change the default preference. Ultimately I may just settle for using the library installed, as I can control that, but it would be a nice feature to have.

Alexanders suggestion is interesting, but I would like to know if there is a cleaner way.

[Edit2]

The original example was simplified. Each module may contain multiple types of imports, e.g. Consumer and Publisher.

Answer 1

I know two method, one is wildly used and another is my guesswork. You can choose one for your situation.

The first one, which is widely used, such as from tornado.concurrent import Future.

try:
    from concurrent import futures
except ImportError:
    futures = None

#define _DummyFuture balabala...

if futures is None:
    Future = _DummyFuture
else:
    Future = futures.Future

Then you can use from tornado.concurrent import Future in other files.

The second one, which is my guesswork, and I write simple demo, but I haven't use it in production environment because I don't need it.

import sys
try:
    import servicelibrary.simple.synchronous
except ImportError:
    import servicelibrary.simple.alternative
    sys.modules['servicelibrary.simple.synchronous'] = servicelibrary.simple.alternative

You can run the script before other script import servicelibrary.simple.synchronous. Then you can use the script as before:

from servicelibrary.simple.synchronous import Publisher
from servicelibrary.simple.synchronous import Consumer

The only thing I wonder is that what are the consequences of my guesswork.

Answer 2

The importlib.import_module might do what you need:

INSTALLED = ['syncronous', 'alternative']  

for mod_name in INSTALLED:
    try: 
        module = importlib.import_module('servicelibrary.simple.' + mod_name)
        Publisher = getattr(module, 'Publisher')

        if Publisher:
            break  # found, what we needed

    except ImportError:
        continue

I guess, this is not the most advance technique, but the idea should be clear. And you can take a look at the imp module as well.

Answer 3

Based on the answers I ended up with the following implementation for Python 2.7.

Examples are simplified for stackoverflow..

from importlib import import_module

PARENT = 'myservicelib.rabbitmq'
MODULES = ['test_adapter',
           'test_two_adapter']
SUCCESS = False

for _module in MODULES:
    try:
        __module = import_module('{0}.{1}'.format(PARENT, _module))
        Consumer = getattr(__module, 'Consumer')
        Publisher = getattr(__module, 'Publisher')
        SUCCESS = True
        break
    except ImportError:
        pass

if not SUCCESS:
    raise NotImplementedError('no supported rabbitmq library installed.')

Although, as I also had some of my projects running Python 2.6 I had to either modify the code, or include importlib. The problem with a production platform is that it isn't always easy to include new dependencies.

This is the compromise I came up with, based on __import__ instead of importlib.

It might be worth checking if sys.modules actually contains the namespace so you don't get a KeyError raised, but it is unlikely.

import sys

PARENT = 'myservicelib.rabbitmq'
MODULES = ['test_adapter',
           'test_two_adapter']
SUCCESS = False

for _module in MODULES:
    try:
        __module_namespace = '{0}.{1}'.format(PARENT, _module)
        __import__(__module_namespace)
        __module = sys.modules[__module_namespace]
        Consumer = getattr(__module, 'Consumer')
        Publisher = getattr(__module, 'Publisher')
        SUCCESS = True
        break
    except ImportError:
        pass

if not SUCCESS:
    raise NotImplementedError('no supported rabbitmq library installed.')

Answer 4

You've got the right idea. Your case works because each subobject has the same sort of classes e.g. both APIs have a class called Publisher and you can just make sure the correct version is imported.

If this isn't true (if possible implementation A and B are not similar) you write your own facade, which is just your own simple API that then calls the real API with the correct methods/parameters for that library.

Obviously switching between choices may require some overhead (i don't know your case, but for instance, let's say you had two libraries to walk through an open file, and the library handles opening the file. You can't just switch to the second library in the middle of the file and expect it to start where the first library stopped). But it's just a matter of saving it:

accessmethods = {}
try:
    from modA.modB import classX as apiA_classX
    from modA.modB import classY as apiA_classY
    accessmethods['apiA'] = [apiA_classX, apiA_classY]
    classX = apiA_classX
    classY = apiA_classY
except:
    pass

try:
    from modC.modD import classX as apiB_classX
    from modC.modD import classY as apiB_classY
    accessmethods['apiB'] = [apiB_classX, apiB_classY]
    classX = apiB_classX
    classY = apiB_classY
except:
    pass

def switchMethod(method):
    global classX
    global classY
    try: 
        classX, classY = accessmethods[method]
    except KeyError as e:
        raise ValueError, 'Method %s not currently available'%method

etc.

Answer 5

A flexible solution, using importlib. This is a complete, working solution that i've tested.

First, the header:

import importlib
parent = 'servicelib.simple'
modules = {'.synchronous':['.alternative', '.alternative_2']}
success = False #an indicator, default is False,
#changed to True when the import succeeds.

We import the required module, set our indicator, and specify our modules. modules is a dictionary, with the key set as the default module, and the value as a list of alternatives.

Next, the import-ant part:

#Obtain the module
for default, alternatives in modules.items():
    try: #we will try to import the default module first
        mod = importlib.import_module(parent+default)
        success = True
    except ImportError: #the default module fails, try the alternatives
        for alt in alternatives:
            try: #try the first alternative, if it still fails, try the next one.
                mod = importlib.import_module(parent+alt)
                success = True
                #Stop searching for alternatives!
                break 
            except ImportError:
                    continue

print 'Success: ', success

And to have the classes, simply do:

Publisher = mod.Publisher
Consumer = mod.Consumer

With this solution, you can have multiple alternatives at once. For example, you can use both rabbitpy and pyAMPQ as your alternatives.

Note: Works with both Python 2 and Python 3.

If you have more questions, feel free to comment and ask!