Symfony2 OAuth keeps giving me a login page when a

I have setup an app with:

FriendsOfSymfony/FOSUserBundle
FriendsOfSymfony/FOSOAuthServerBundle
FriendsOfSymfony/FOSRestBundle

I have successfully created a client and can get an access token using a url like this

http://api.mydomain.com/oauth/v2/token?client_id=CLIENT_ID&client_secret=CLIENT_SECRET&grant_type=password&username=mikebates&password=secret

However, when I then try to access the a url like this http://api.mydomain.com/api/surgeries/45/details?access_token=ACCESS_TOKEN

I get redirected to the symfony login page, but I can't have that or the mobile app I'm build to consume this REST API won't be able to get access.

I used this tutorial to set things up (removing the User / UserRepository classes in there and altering it to work with FOSUserBundle) http://blog.tankist.de/blog/2013/07/16/oauth2-explained-part-1-principles-and-terminology/

I'm not sure where I have told symfony to redirect to the login page, I want to change that logic to just authenticate from the token.

This is an overview of my setup

security.yml

security:
    encoders:
        FOS\UserBundle\Model\UserInterface: sha512

    role_hierarchy:
        ROLE_ADMIN:       ROLE_USER
        ROLE_SUPER_ADMIN: ROLE_ADMIN

    providers:
        fos_userbundle:
            id: fos_user.user_provider.username  

    firewalls:
        ...

        login:
            pattern:  ^/secured/login$
            security: false

        oauth_token:
            pattern:    ^/oauth/v2/token
            security:   false

        oauth_authorize:
            pattern:    ^/oauth/v2/auth
            form_login:
                provider: fos_userbundle
                check_path: _security_check
                login_path: _demo_login
            anonymous: true

        api:
            pattern:    ^/api
            fos_oauth:  true
            stateless:  true
            anonymous:  false # can be omitted as its default value

    access_control:
        - { path: ^/login$, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/register, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/resetting, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/api, roles: [ IS_AUTHENTICATED_FULLY ] }
        - { path: ^/oauth/v2/auth_login, role: IS_AUTHENTICATED_ANONYMOUSLY }

config.yml

# FriendsOfSymfony : Rest
fos_rest:
    disable_csrf_role: ROLE_API
    param_fetcher_listener: true
    view:
        view_response_listener: 'force'
        formats:
            xml:  true
            json: true
        templating_formats:
            html: true
        jsonp_handler: ~
    format_listener:
        rules:
            - { path: ^/, priorities: [ html, jsonp, json, xml ], fallback_format: ~, prefer_extension: true }
    exception:
        codes:
            'Symfony\Component\Routing\Exception\ResourceNotFoundException': 404
            'Doctrine\ORM\OptimisticLockException': HTTP_CONFLICT
        messages:
            'Symfony\Component\Routing\Exception\ResourceNotFoundException': true
    allowed_methods_listener: true
    access_denied_listener:
        json: true
    body_listener: true

# FriendsOfSymfony : User
fos_user:
    db_driver: orm
    firewall_name: main
    user_class: Incompass\UserBundle\Entity\Person
    group:
        group_class: Incompass\UserBundle\Entity\Group

# FriendsOfSymfony : OAuth
fos_oauth_server:
    db_driver: orm
    client_class: Incompass\AuthBundle\Entity\Client
    access_token_class: Incompass\AuthBundle\Entity\AccessToken
    refresh_token_class: Incompass\AuthBundle\Entity\RefreshToken
    auth_code_class: Incompass\AuthBundle\Entity\AuthCode
    service:
        user_provider: fos_user.user_provider.username
        options:
            supported_scopes: user

I solved this by changing the order of the firewalls in security.yml

firewalls:
    oauth_authorize:
        pattern:    ^/oauth/v2/auth
        form_login:
            provider: fos_userbundle
            check_path: /oauth/v2/auth_login_check
            login_path: /oauth/v2/auth_login
        anonymous: true

    oauth_token:
        pattern:    ^/oauth/v2/token
        security:   false

    api:
        pattern:    ^/api
        fos_oauth:  true
        stateless:  true

    login:
        pattern:  ^/secured/login$
        security: false

    main:
        pattern: ^/
        form_login:
            provider: fos_userbundle
            csrf_provider: form.csrf_provider
        logout:       true
        anonymous:    true

    dev:
        pattern:  ^/(_(profiler|wdt)|css|images|js)/
        security: false

    secured_area:
        pattern:    ^/secured/
        form_login:
            provider: fos_userbundle
            check_path: _security_check
            login_path: _demo_login
        logout:
            path:   _demo_logout
            target: _demo