nginx proxy_pass leads to 404 Not Found page

Question

I have a angular application running in a docker ubuntu image that has nginx installed. I want to deploy this image to Kubernetes and use a nginx proxy to redirect all calls to /api to my backend service in Kubernetes.

My static web resources lie in /var/www/html and I add the following config to /etc/nginx/conf.d:

upstream backend-service {
  server backend-service:8080;
}

server {
  listen 80;

  location / {
    try_files $uri $uri/ /index.html;
  }

  location ^~ /api {
    proxy_pass http://backend-service;
  }
}

Accessing the frontend service on / or /#/dashboard returns the expected component of my Angular page, but a call to /api/v1/data only shows the default nginx 404 Not Found page.

What do I need to modify to have my backend calls redirected to my backend?

I use nginx 1.10.3 on ubuntu 16.04 and my frontend Dockerfile looks like this:

FROM ubuntu:16.04

# Install curl, nodejs and nginx
RUN apt-get update && \
  apt-get install -y curl && \
  curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
  apt-get install -y nodejs nginx && \
  rm -rf /var/lib/apt/lists/*

# Create directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Copy and build rest of the app
COPY . /usr/src/app
RUN npm install
RUN node_modules/@angular/cli/bin/ng build --prod
RUN cp -a dist/. /var/www/html

# Configure and start nginx
COPY frontend.conf /etc/nginx/conf.d

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

Edit: Information about backend-service

The backend service listens to get and post requests on /api/v1/data and is reachable in Kubernetes via a Service named backend-service.

Edit2: Nginx access.log

https://gist.github.com/Steffen911/a56e3175bf12e511048d01359a475724

172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET / HTTP/1.1" 200 380 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /styles.d41d8cd98f00b204e980.bundle.css HTTP/1.1" 200 0 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /inline.c9a1a6b995c65c13f605.bundle.js HTTP/1.1" 200 1447 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /polyfills.117078cae3e3d00fc376.bundle.js HTTP/1.1" 200 97253 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /main.3e9a37b4dd0f3bf2465f.bundle.js HTTP/1.1" 200 64481 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /vendor.146173c1a99cc2172a5f.bundle.js HTTP/1.1" 200 661261 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /api/v1/data/ HTTP/1.1" 404 209 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/home.jpg HTTP/1.1" 200 2608 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/busy.gif HTTP/1.1" 200 48552 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/background_light.png HTTP/1.1" 200 170599 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/google.svg HTTP/1.1" 200 2232 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/email.svg HTTP/1.1" 200 1596 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /favicon.ico HTTP/1.1" 200 198 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:44 +0000] "GET /api/v1/data/ HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"

The error.log file is empty.

Edit3: New nginx version and other SO thread

I also tried nginx 1.12.1 and it shows the same behaviour. The answers to this question also haven't helped: nginx proxy_pass 404 error, don't understand why

Edit4: I uploaded a minimal example that reproduces my problem on GitHub

https://github.com/Steffen911/nginx-sample

Answer 1

From your troubleshooting of nginx, it appears that the nginx configuration file you have has effectively no effect — you report getting 404 Not Found errors for everything other than the index page, with all the directives, from try_files in location /, to proxy_pass and return 200 test in a more specific location ^~ /api, having no effect.

As such, the problem appears to be in the Dockerfile — it appears that most other NGINX + Docker tutorials remove default configurations (e.g., with RUN rm /etc/nginx/conf.d/default.conf), whereas your file is missing any such removal.

In fact, Debian/Ubuntu appear to have the non-standard directories of questionable utility called /etc/nginx/sites-available and /etc/nginx/sites-enabled, which, by default, must contain a default file with a presumptuous listen 80 default_server, effectively taking precedence over any other listen of the same port in the absence of a more specific server_name.

---

As such, there are multiple independent solutions:

---

• Do not use fundamentally broken packages like those offered by Debian/Ubuntu. I once spent a good amount of time pulling my hair trying to figure out why my configs don't work, only to notice that even the backup files from emacs like test.conf~ get included in Debian through Debian's default include /etc/nginx/sites-enabled/*;. Sites-Enabled Is Evil.

  Note that NGINX provides official binary packages for most distributions, which wouldn't have had this issue, as it doesn't try to define a default_server in its /etc/nginx/conf.d/default.conf, instead doing a listen 80; with server_name localhost;, getting out of your way automatically by itself in most circumstances.

  E.g., replace FROM ubuntu:16.04 with FROM nginx in your Dockerfile to be using NGINX official image.

---

• If still using nginx from Debian/Ubuntu, make sure to RUN rm /etc/nginx/sites-enabled/default in your Dockerfile to remove the default_server listen.

---

• Use the server_name directive to define hostname-based servers, possibly together with the listen directive with the default_server parameter, too.

  Note that a duplicate server_name specification results in a configuration warning (with the [warn] severity), but a duplicate default_server is a configuration error ([emerg] severity), which might help troubleshoot the issue earlier.

Answer 2

a call to /api/v1/data only shows the default nginx 404 Not Found page

The proxy_pass directive is highly unlikely to be generating the default nginx 404 Not Found error page itself.

• The 404 might be generated by the upstream, in which case, absent further instructions, nginx will simply propagate the message from upstream. If you get an nginx signature in your 404, then it means that the upstream is also running nginx, possibly to your surprise, revealing the configuration culprit.

• If the 404 is actually generated by nginx indeed, then it might be the case that the location doesn't match. Try putting return 200 thisisatest; in place of proxy_pass to troubleshoot the issue.

  However, in your specific case, location ^~ /api { is an almost-impossible directive to not match a /api/v1/data/ request URI — the only thing I could think of is if you possibly have a try_files in your server config outside of any other location directive, which could then possibly make all location directives null and void. Are you sure your try_files is fully contained within the location / context?