Using @RabbitListener with Jackson2JsonMessageConv

I have spent the last little while trying to get it so that my handler that is registered using @RabbitListener will have the message converted with the Jackson2JsonMessageConverter. Unfortunately no matter what configuration setup I try only the SimpleMessageConverter is used.

@Component
@Slf4j
public class TestQueueListener {
    @RabbitListener(queues = "#{@allQueue}")
    public void processMessage(String data) {
        log.trace("Message received: {}", data);
    }

    @RabbitListener(queues = "#{@invokeQueue}")
    public void processSpawnInstanceMessage(TestConfig config) {
        log.trace("Invoking with config: {}", config);

        invokeSomeMethod(config);
    }
}

This is the config I have currently that I think is closest to what is documented:

@Configuration
@EnableRabbit
public class MessagingConfiguration {
    @Bean
    public MessageConverter messageConverter() {
        ContentTypeDelegatingMessageConverter messageConverter = new ContentTypeDelegatingMessageConverter();
        messageConverter.addDelgate("application/json", new Jackson2JsonMessageConverter());
        return messageConverter;
    }

    @Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(messageConverter);
        return rabbitTemplate;
    }

    @Bean
    public TopicExchange testExchange() {
        return new TopicExchange("test");
    }

    @Bean
    public Queue allQueue() {
        return new Queue("all", false, true, true);
    }

    @Bean
    public Binding allBinding(TopicExchange testExchange, Queue allQueue) {
        return BindingBuilder.bind(allQueue).to(testExchange).with("*");
    }

    @Bean
    public Queue invokeQueue() {
        return new Queue("invoke", false, true, true);
    }

    @Bean
    public Binding invokeBinding(TopicExchange testExchange, Queue invokeQueue) {
        return BindingBuilder.bind(invokeQueue).to(testExchange).with("invoke");
    }
}

What I understand so far, which is where I am stuck, is that RabbitTemplate is used when sending a message from the application to RabbitMQ. As such I sort of understand why my custom MessageConverter is not being used for the Handler invocation. Unfortunately I don't see how to set the MessageConverter on the incoming side. How do I use the MessageConverter that I have configured with the incoming messages?

See the documentation about how to configure the listener containers created for annotated endpoints:

@Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
    SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
    factory.setConnectionFactory(rabbitConnectionFactory());
    factory.setMessageConverter(messageConverter());
    return factory; 
}