How can PHP driven API authenticate genuine client

Using PHP, how do you securely authenticate an API call, cross-domain, with the following criteria:

Must be called from a given domain.com/page (no other domain)
must have a given key

Some background: Please read carefully before answering...

My web application will display a javascript widget on a client's website via a call like the one below. So we're talking about cross-domain authentication for a script to be served but only to genuine client and for a given URL only!

At the moment the widget can be included by the CLIENT's website, by a single-line of javascript.

Example client-website.com/page/with/my-widget

<head>
...
<script src="//ws.my-webappserver.com/little-widget/?key=api_key"></script>
...
</head>

Now, in reality this does not call javascript directly but a PHP script on my remote server which sits in front of the actual javascript for the purpose of doing some authentication.

The PHP script behind the above call does this first:

checks that API key ($_REQUEST["key"]) matches the user's record in the database.
checks the referrer's URL ($_SERVER['HTTP_REFERER']) against the a record in the database***.

This is simplified, but in essence the server looks like:

if ($_SERVER['HTTP_REFERER'] !== "http://client-website.com/page/with/my-widget")
  && $_REQUEST["Key"] == "the_long_api_key") {
    header("Content-type: application/x-javascript");
    echo file_get_contents($thewidgetJS_in_secret_location_on_server);
} else {
  //pretend to be a 404 page not found or some funky thing like that
}

***The widget is only allowed to run on CERTAIN websites/pages

Now, here's the problem:

the key is on the client side so anyone can view source and get the key
referrers can be spoofed (by cURL) for example

And a little snag: I can't tell the clients to stick the key inside a php script on their server because they could be running any server side language!

So how can I make my web service secure and only accessible by a given domain/page?

Any help would be really appreciated!

ps: The widget does some uploading to a kind of drop box - so security is key here!

id recommend using a whitelist approach to this issue, i am not entirely sure this will solve the problem however i have used a similar technique for a payment processor which requires you to whitelist server-ips.