Perl REST::Client - Garbage data in response

Question

I'm having issues getting a valid response from Perl REST::Client against Red Hat Satellite REST API. I'm getting the following encoded response:

$VAR1 = '���j�0��~
�g9��   ���#�9�`dIm�m�-���uJ
        �����f4U�@▒��
                     ���F��xګ X�;�\'r��/���3R�s�C�u�*�2_N��٧�������f\\�������WA0����نp��T͖�l�▒Pȣ}�x��8�&�d�n��ߦ`��.���Tƙ�V�c�&����a���%�ZH·�aJ�0�yT��q� �Jz��ճMO�\\�����'

And when I decode it via Encode::decode_utf8, it likewise appears to be garbled:

$VAR1 = "\x{fffd\x{fffd}\x{fffd}\x{fffd}j\x{fffd}0\x{fffd}\x{fffd}~
\x{fffd}g9\x{fffd}\x{fffd}      \x{fffd}\x{fffd}\x{fffd}#\x{fffd}9\x{fffd}`dIm\x{fffd}m\x{fffd}-\x{fffd}\x{fffd}\x{fffd}uJ
        \x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}f4U\x{fffd}\@▒\x{fffd}\x{1a0220}
                                                                                \x{fffd}\x{fffd}\x{fffd}F\x{fffd}\x{fffd}x\x{6ab} X\x{fffd};\x{fffd}'r\x{fffd}\x{fffd}/\x{fffd}\x{fffd}\x{fffd}3R\x{fffd}s\x{fffd}C\x{fffd}u\x{fffd}*\x{fffd}2_N\x{fffd}\x{fffd}\x{667}\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}f\\\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}WA0\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{646}p\x{fffd}\x{fffd}T\x{356}\x{fffd}l\x{fffd}▒P\x{223}}\x{fffd}x\x{fffd}\x{fffd}8\x{fffd}&\x{fffd}d\x{fffd}n\x{fffd}\x{fffd}\x{7e6}`\x{fffd}\x{fffd}.\x{fffd}\x{fffd}\x{fffd}T\x{199}\x{fffd}V\x{fffd}c\x{fffd}&\x{fffd}\x{fffd}\x{fffd}\x{fffd}a\x{fffd}\x{fffd}\x{fffd}%\x{fffd}ZH\x{b7}\x{fffd}aJ\x{fffd}0\x{fffd}yT\x{fffd}\x{fffd}q\x{fffd} \x{fffd}Jz\x{fffd}\x{fffd}\x{573}MO\x{fffd}\\\x{fffd}\x{fffd}\x{fffd}\x{fffd}\x{fffd}";

The script I am using to test:

#!/usr/bin/perl

use strict;
use warnings;

use 5.010;
use REST::Client;
use Data::Dumper;

require JSON;
require MIME::Base64;
require HTTP::Cookies;

my $host        = 'https://sat.example.com/api/v2/domains';
my $client      = REST::Client->new({useragent => LWP::UserAgent->new(cookie_jar => HTTP::Cookies->new)});
my $headers     = {
        'Authorization'         => 'Basic XXXXXXXXX',
        'Accept-Encoding'       => scalar HTTP::Message::decodable,
        #'Content-Type'         => 'application/json',
        'Content-Type'          => 'application/json;charset=utf8',
        'Connection'            => 'keep-alive',
        'Accept'                => 'application/json',
        'Host'                  => URI->new($host)->canonical->host_port,
        #'Charset'              => 'UTF-8'
};

$client->addHeader($_, $headers->{$_}) for keys %{$headers};
$client->setCa('/var/www/html/pub/katello-server-ca.crt');

print Dumper JSON::decode_json($client->GET($host)->responseContent);

#print Dumper $client->GET($host);
__END__

When I look at the REST::Client object via Data::Dumper all headers appear to be set properly and are readable. The only thing that is not readable is the actual content.

I am able query the API using curl commands from the same server:

$ curl -\# -X GET -v -H 'Accept: application/json' -H 'Authorization: Basic XXXXXXXXX' --cacert katello-server-ca.crt https://sat.example.com/api/v2/domains -o /dev/null 
* About to connect() to sat.example.com port 443 (#0)
*   Trying 192.168.0.1...
* Connected to sat.example.com (192.168.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: katello-server-ca.crt
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=sat.example.com,OU=SomeOrgUnit,O=Katello,ST=North Carolina,C=US
*       start date: Oct 11 20:24:12 2018 GMT
*       expire date: Jan 17 20:24:12 2038 GMT
*       common name: sat.example.com
*       issuer: CN=sat.example.com,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North Carolina,C=US
> GET /api/v2/domains HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sat.example.com
> Accept: application/json
> Authorization: Basic XXXXXXXXX
> 
< HTTP/1.1 200 OK
< Date: Wed, 24 Apr 2019 18:50:34 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux)
< Foreman_version: 1.15.6.48
< Foreman_api_version: 2
< Apipie-Checksum: 2a54cbc5a3f59fad6e7e697ec609cda8
< Cache-Control: max-age=0, private, must-revalidate
< X-Request-Id: 57672b22-6a4f-4b3a-83fb-6b1a54747d67
< X-Runtime: 0.036638
< Content-Security-Policy: default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data: *.gravatar.com; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'
< Strict-Transport-Security: max-age=631152000; includeSubdomains
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: sameorigin
< X-Permitted-Cross-Domain-Policies: none
< X-XSS-Protection: 1; mode=block
< X-Powered-By: Phusion Passenger 4.0.18
< Set-Cookie: _session_id=d35c9df0bce80baeca85b0ad38298ee8; path=/; secure; HttpOnly
< ETag: W/"0eeb7a2c131e780c72202cf712c972e3"
< Status: 200 OK
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=utf-8
< 
{ [data not shown]
######################################################################## 100.0%* Connection #0 to host sat.example.com left intact

I've used REST::Client against may different API's with no issue, this is the first API that has really given me any grief. I'm not sure where to start troubleshooting after decoding the response. Any suggestions would be helpful.

Output of print Dumper $client->GET($host)->{_res}->as_string;

$VAR1 = 'HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Connection: close
Date: Thu, 25 Apr 2019 19:45:48 GMT
ETag: W/"0eeb7a2c131e780c72202cf712c972e3-gzip"
Server: Apache/2.4.6 (Red Hat Enterprise Linux)
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 242
Content-Type: application/json; charset=utf-8
Apipie-Checksum: 2a54cbc5a3f59fad6e7e697ec609cda8
Client-Date: Thu, 25 Apr 2019 19:45:53 GMT
Client-Peer: 192.168.0.1:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=sat.example.com
Client-SSL-Cert-Subject: /C=US/ST=North Carolina/O=Katello/OU=SomeOrgUnit/CN=sat.example.com
Client-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Content-Security-Policy: default-src \'self\'; child-src \'self\'; connect-src \'self\' ws: wss:; img-src \'self\' data: *.gravatar.com; script-src \'unsafe-eval\' \'unsafe-inline\' \'self\'; style-src \'unsafe-inline\' \'self\'
Foreman_api_version: 2
Foreman_version: 1.15.6.48
Set-Cookie: _session_id=ea51c42b7a65478d27b21c2fb02e4c4a; path=/; secure; HttpOnly
Status: 200 OK
Strict-Transport-Security: max-age=631152000; includeSubdomains
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: sameorigin
X-Permitted-Cross-Domain-Policies: none
X-Powered-By: Phusion Passenger 4.0.18
X-Request-Id: 528535f3-c8cf-4d8b-91db-1572e94a974a
X-Runtime: 4.600748
X-XSS-Protection: 1; mode=block

???j?0??~
?g9??   ???#?9?`dIm?m?-???uJ
    ???f4U?@?????
                     ???F??xګ X?;?\'r??/??3R?s?C?u?*?2_N??٧??????f\\???????WA0????نp??T͖?l?Pȣ}?x??8?&?d?n??ߦ`??.???Tƙ?V?c?&????a???%?ZH·?aJ?0?yT?q? ?Jz??ճMO?\\?????
';

Answer 1

Based on the comments attached to your question, we can say the problem is that the server is responding with gzip compressed data, and the REST::Client package simply returns the raw data without decoding it first. You can access the underlying HTTP::Response object directly using $client->{_res} on your REST::Client object, and you can get the properly unzipped and charset-converted data using the ->decoded_content method on that. Note that you'll need to use JSON::from_json() rather than JSON::decode_json() on the decoded content, because it's already been converted from a UTF-8 byte sequence to a Unicode string as part of the decoding.

In short, then, what you want is the following.

print Dumper JSON::from_json($client->GET($host)->{_res}->decoded_content);

Directly accessing the underlying object isn't pretty, but unless you want to patch REST::Client, it's the easiest way to get the result you want.