Setting the RPATH for external projects?

I'm trying to setup a project where my repository imports all dependencies as git submodules for easy development. My colleagues can simply clone the repo, git submodule update --init --recursive, cmake . and make and have a fully working dev environment in place. The directory structure is setup as a superbuild with a CMakeLists.txt at the top level that builds all the submodules using ExternalProject_Add, resulting in the following structure:

root
 - CMakeLists.txt (superbuild)
 - git_submodule_1
 - git_submodule_2
 - usr
    - lib
    - include
 - my_project
    - CMakeLists.txt (project)

The CMakeLists.txt looks something like this:

SET (INSTALL_PREFIX ${CMAKE_SOURCE_DIR}/usr)

# Install git_submodule_1 with ${INSTALL_PREFIX} as a prefix
ExternalProject_Add( ... )

# Install git_submodule_2 with ${INSTALL_PREFIX} as a prefix
ExternalProject_Add( ... )

ExternalProject_Add(
    MyProject
    PREFIX ${CMAKE_SOURCE_DIR}/my_project
    DEPENDS ExternalProject_git_submodule_1 ExternalProject_git_submodule_2

    SOURCE_DIR ${CMAKE_SOURCE_DIR}/my_project
    CMAKE_ARGS
      -DCMAKE_LIBRARY_PATH:string=${INSTALL_PREFIX}/lib
      -DCMAKE_PROGRAM_PATH:string=${INSTALL_PREFIX}/bin
      -DCMAKE_INCLUDE_PATH:string=${INSTALL_PREFIX}/include

    # etc, nothing nonstandard here
)

The build process works great. I make at the top level, dependencies are installed into usr, I cd into my_project, I do my work, all the built shared libraries are found and linked, I'm happy.

However, when I go to run an executable on OS X built inside my_project, I find that the dynamic libs placed into the usr/lib directory cannot be found. It appears that CMake only sets the RPATH for libraries built within the project directory, which in this case is just my_project.

Is there any way I can add the custom install location to the the RPATH for build-time libraries and executables?

A few notes:

This issue only appears to affect OS X. Linux doesn't exhibit these problems at all.
Setting the DYLD_LIBRARY_PATH to include the custom install location works. However, this adds an additional step to the setup, and it gets annoying when I try to debug installation issues.
Setting the DYLD_FALLBACK_LIBRARY_PATH also works, although that's also not a good option because it also adds an additional set, and homebrew users won't like this option.

For the executables and shared libraries built with my_project/CMakeLists.txt add ${INSTALL_PREFIX}/lib to the install rpath and also make CMake link the targets in the build tree with that install rpath in the following way:

set_target_properties(my_exe PROPERTIES INSTALL_RPATH "${INSTALL_PREFIX}/lib")
set_target_properties(my_exe PROPERTIES BUILD_WITH_INSTALL_RPATH ON)

That way the loader should find the external libraries installed to the ${CMAKE_SOURCE_DIR}/usr/lib directory upon running an executable from the build tree.