HTTPS GZIP NGINX WTF

I've been trying for days now to get nginx to serve gzip content mainly because googles speed test told me to do it and we are trying to increase our SEO. I can't for the life of me understand what is going wrong here:

We are behind a firewall and serve two web heads underneath a load balancer. No matter what I've tried I can not get the response headers to come back with content-encoding:gzip. However, when I make the request using curl I can. Also when I access the site via https I do get the response back with gzip however it has nothing to do with nginx as I've turned gzip off in nginx and I still get the same response. What else would be serving the content as gzip?

UPDATE

Ok sorry after posting this several times with no response I was a little frustrated. Here is some info to go off:

We are using nginx 1.8.0 with php-fpm. The site is a Magento framework. I am trying to serve the main html pages compressed with gzip as well as the included css/javascript files. These files currently show not compressed in the response headers and Google pagespeed also says they are not compressed. Here is an example response header I see

Cache-Control:max-age=31536000
Connection:keep-alive
Content-Type:application/x-javascript; charset=utf-8
Date:Tue, 15 Mar 2016 15:15:13 GMT
ETag:"pub1448944926;gz"
Expires:Wed, 15 Mar 2017 15:15:13 GMT
Keep-Alive:timeout=8
Last-Modified:Tue, 01 Dec 2015 04:42:06 GMT
Server:nginx
Transfer-Encoding:chunked
Vary:Accept-Encoding

When I request the page via curl I get the gzip content.

curl -I -H 'Accept-encoding:gzip' mysite.com

We have removed the staging site from the load balancer and the issue persists; eliminating any issues (for now) that may come form the load balancer.

When I access the site via https I get gzip content and here is the response header

Cache-Control:max-age=31536000
Connection:keep-alive
Content-Encoding:gzip
Content-Length:67085
Content-Type:application/x-javascript; charset=utf-8
Date:Tue, 15 Mar 2016 15:51:31 GMT
ETag:"pub1448944926;gz"
Expires:Wed, 15 Mar 2017 15:51:31 GMT
Keep-Alive:timeout=8
Last-Modified:Tue, 01 Dec 2015 04:42:06 GMT
Server:nginx
Vary:Accept-Encoding

Here are the relative config files for nginx

nginx.conf

user                    nginx;
worker_processes        4;
pid                     /var/run/nginx.pid;
error_log               /var/log/nginx/error.log;

events {
    worker_connections  1024;
    multi_accept        on;
    use                 epoll;
}

http {
    include             /etc/nginx/mime.types;
    charset             utf-8;
    default_type        application/octet-stream;

    #access_log         /var/log/nginx/access.log main;
    access_log          off;

    log_format main     '$remote_addr - $remote_user [$time_local]     "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';

    # compression
    gzip                on;
    gzip_http_version   1.0;
    gzip_vary           on;
    gzip_comp_level     5;
    gzip_proxied        any;
    gzip_min_length    100;
    #   gzip_min_length     10240;
    gzip_buffers        16 8k;
    gzip_types          text/plain text/css application/x-javascript     text/comma-separated-values text/xml application/xml application/xml+rss     application/atom+xml text/javascript;
    #gzip_disable       "MSIE [1-6].(?!.*SV1)";

    # general options
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         off;
    autoindex           off;
    server_tokens       off;
    merge_slashes       on;
    client_header_buffer_size           1k;
    client_body_buffer_size             32k;
    client_max_body_size                64m;
    server_names_hash_bucket_size       128;
    large_client_header_buffers         2 1k;

    # timeouts
    send_timeout                        10;
    keepalive_timeout                   2 8;
    keepalive_requests                  200;
    client_body_timeout                 12;
    client_header_timeout               12;
    reset_timedout_connection           on;

    # pass through from load balancer
    real_ip_header X-Forwarded-For;
    set_real_ip_from 0.0.0.0/0;

    # detect https
    map $scheme $fastcgi_https {
        default "";
        https on;
    }

    # PHP-FPM
    upstream phpfpm {
        server unix:/run/php-fpm/php-fpm.sock weight=1 max_fails=5     fail_timeout=10;
    }

    # include active sites
    include /etc/nginx/sites-enabled/*;
    server {
        listen 80 spdy default_server;
        root /var/www/mysite.com;
        location ^~ /app/                       { return 403; }
        location ^~ /includes/                  { return 403; }
        location ^~ /media/downloadable/        { return 403; }
        location ^~ /pkginfo/                   { return 403; }
        location ^~ /report/config.xml          { return 403; }
        location ^~ /var/                       { return 403; }
        location ^~ /lib/                       { return 403; }
        location ^~ /dev/                       { return 403; }
        location ^~ /RELEASE_NOTES.txt          { return 403; }
        location ^~ /downloader/pearlib         { return 403; }
        location ^~ /downloader/template        { return 403; }
        location ^~ /downloader/Maged           { return 403; }
        location ~* ^/errors/.+\.xml            { return 403; }
    }

}

Sites-enabled/mysite

server {

    #mysiteip is an actual ip that I've removed for security

    listen mysiteip:80;  
    server_name www.test.mysite.com;
    return 301 $scheme://test.mysite.com$request_uri;
}
server {
    # settings
    listen mysiteip:80;
    listen mysiteip:443 ssl;
    server_name test.mysite.com;
    root /var/www/mysite.com/testing/current/;
    index index.html index.htm index.php;

    # security
    ssl_protocols TLSv1.2;
    ssl_ciphers RC4:HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    # SSL Certificate Settings
    ssl_certificate     /etc/nginx/ssl/bundle.crt;
    ssl_certificate_key /etc/nginx/ssl/star_mysite_com.key;


    access_log /var/log/nginx/mysite.access.log;
    error_log /var/log/nginx/www-mysite-com_error.log;

    # routes
    include /etc/nginx/conf.d/security.conf;
    include /etc/nginx/conf.d/assets.conf;
    include /etc/nginx/conf.d/rewrites.conf;

    # Attempt to serve the request by trying direct file, directory, Magento front controller

    large_client_header_buffers 8 16k;

    location / {
        try_files $uri $uri/ /index.php?$args;
        expires max;
    }

    # The downloader has its own index.php that needs to be used
    location ~* ^(/downloader)(.*) {
        try_files $uri $uri/ /downloader/index.php$1;
    }

    # REST API endpoint
    location /api {
        rewrite ^/api/rest /api.php?type=rest last;
        rewrite ^/api/v2_soap /api.php?type=v2_soap last;
        rewrite ^/api/soap /api.php?type=soap last;
    }

    # Pass PHP scripts to PHP-FPM daemon
    location ~* \.php$ {
        # filter out problem conditions
        location ~ \..*/.*\.php$ { return 404; }

        # bring in parameters
        include /etc/nginx/conf.d/fastcgi.conf;
        fastcgi_param MAGE_RUN_CODE default;
        fastcgi_param MAGE_RUN_TYPE store;

        # DEVELOPER MODE
        #fastcgi_param MAGE_IS_DEVELOPER_MODE true;

        # send requests to upstream, but blacklist media location from fcgi
        if ($uri !~ "^/(media)/") {
            fastcgi_pass phpfpm;
        }
    }
}

conf.d/rewrites.conf

# I am using this rewrite for the fooman speedster extension

location /skin/m {
   rewrite ^/skin/m/([^/]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1;
}

Any insight as to what is going on or being done wrong would be greatly appreciated. I can provide more info if needed. Also the rewrite for Fooman is used for minification and even without this extension installed and the rewrite removed I still get no gzip.

回答1:

Your HTTP responses are probably cached. These HTTP headers give it away:

Cache-Control:max-age=31536000
Expires:Wed, 15 Mar 2017 15:15:13 GMT

This basically tells everyone who requests the response to cache the response as they see fit, because it will be valid and cacheable until March 2017. In your example this is for Javascript, so it's fine, but if you test the resource again, then it is quite possible that the old response is being returned. You would need to make sure every cache is emptied and no proxy has cached the response. Otherwise you may be getting the response from an intermediate location, not your Nginx server, so you cannot verify any GZIP settings in that case.

HTTPS requests are generally not cached by any proxy (because they do not see the content), so that may be the reason why it works there.

The following lines in your config probably leads to accidental caching:

location / {
    try_files $uri $uri/ /index.php?$args;
    expires max;
}

Nginx adds expires and max-age cache control headers to all requests by using the expires directive here, even dynamic requests. You should check if your HTML/PHP pages aren't cached by using this header and if that may be part of the problem.

回答2:

Cant add comment now because of Bounty. @iquiot @iquito Thanks I disabled expires max and no luck. I tracked down to this being set in the Fooman extension. and set max-age to zero. Here is the new response header ... but still no gzip

Cache-Control   max-age=0
Connection  keep-alive
Content-Type    text/css; charset=utf-8
Date     Thu, 17 Mar 2016 19:35:05 GMT
Etag    "pub1457625059;gz"
Last-Modified Thu, 10 Mar 2016 15:50:59 GMT
Server nginx/1.8.0
Transfer-Encoding chunked
Vary    Accept-Encoding

UPDATE:

I removed myself from the larger picture to try and debug this issue. I created a test.html file and am using this to try and see if I can get any sort of compression going. One strange thing that I am noticing is that if I set the gzip_min_length to say 100 I get content-length in the header when the content is under 100 however as soon as I exceed 100 the content-length goes away.

UPDATE 2: When using curl any sort of "Accept:" header that is set does not return gzip in the response. Could this be a possible culprit?

UPDATE 3: using this little tool for firefox https://addons.mozilla.org/en-US/firefox/addon/modify-headers/developers I was able to modify the headers sent. By completely disabling the user agent header I now get the proper response header with content-encoding gzip. However I do not have anywhere in my nginx config a disable parameter set for gzip which would give such behavior.