NGINX using $server_port in upstream

Question

I want to proxy_pass to an upstream that has one host but multiple ports. This is what I currently have:

upstream myUps {
  zone myUps 32k;
  server myUps.hostname.com:$server_port;
}

When I try this configuration and reload nginx, I get the following:

"invalid port in upstream"

The basic idea is simple, I just want to keep the port that is passed in to the server (server_port), and use it when I pass on to the upstream. I can hard code port numbers in the upstream like this:

...
server myUps.hostname.com:1234;
...

And this will work every time for that specific port. But I need to be able to use multiple ports.

EDIT - This is what my proxy_pass looks like:

proxy_pass http://myUps;
-or-
proxy_pass http://myUps:$server_port;

I do not know if the second one is right, I am just playing with it to see if its possible to pass the port number from the location to the upstream.

Other things Ive tried that didnt work:

Setting a parameter in a map acting as a global variable. Specifying the port from the "location" section of the server. Set up a hash (Only works for IP so doesnt really help here.).