aws chalice ignoring proxy config

Question

I am trying to deploy the helloworld example using chalice behind a corporate proxy. On my machine I use cntlm to proxy the requests with my credentials to the corp proxy.

I am monitoring cntlm logs but when I run chalice deploy the request never hits my local proxy.

$ echo $http_proxy $https_proxy
http://localhost:3128/ http://localhost:3128/
$ chalice --version
chalice 1.6.0, python 3.6.5, linux 4.15.0-24-generic
$ chalice --debug deploy
Creating deployment package.
2018-07-29 13:46:52,813 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,813 - botocore.session - DEBUG - Loading variable config_file from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable region from config file with value 'us-east-1'.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable data_path from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable config_file from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,817 - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2018-07-29 13:46:52,818 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,818 - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2018-07-29 13:46:52,818 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,818 - botocore.credentials - DEBUG - Looking for credentials via: env
2018-07-29 13:46:52,818 - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2018-07-29 13:46:52,818 - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2018-07-29 13:46:52,818 - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2018-07-29 13:46:52,818 - botocore.credentials - DEBUG - Looking for credentials via: config-file
2018-07-29 13:46:52,819 - botocore.credentials - INFO - Credentials found in config file: ~/.aws/config
2018-07-29 13:46:52,819 - botocore.loaders - DEBUG - Loading JSON file: /home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/data/endpoints.json
2018-07-29 13:46:52,822 - botocore.session - DEBUG - Loading variable profile from defaults.
2018-07-29 13:46:52,822 - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f8ea98d59d8>
2018-07-29 13:46:52,827 - botocore.loaders - DEBUG - Loading JSON file: /home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/data/iam/2010-05-08/service-2.json
2018-07-29 13:46:52,836 - botocore.hooks - DEBUG - Event creating-client-class.iam: calling handler <function add_generate_presigned_url at 0x7f8ea99290d0>
2018-07-29 13:46:52,837 - botocore.regions - DEBUG - Using partition endpoint for iam, us-east-1: aws-global
2018-07-29 13:46:52,837 - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2018-07-29 13:46:52,840 - botocore.endpoint - DEBUG - Setting iam timeout as (60, 60)
2018-07-29 13:46:52,840 - botocore.loaders - DEBUG - Loading JSON file: /home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/data/_retry.json
2018-07-29 13:46:52,841 - botocore.client - DEBUG - Registering retry handlers for service: iam
2018-07-29 13:46:52,841 - botocore.hooks - DEBUG - Event before-parameter-build.iam.GetRole: calling handler <function generate_idempotent_uuid at 0x7f8ea98f5d90>
2018-07-29 13:46:52,841 - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetRole) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-chalice/1.6.0 Python/3.6.5 Linux/4.15.0-24-generic exec-env/AWS_Lambda aws-chalice/1.6.0 Botocore/1.10.65'}, 'body': {'Action': 'GetRole', 'Version': '2010-05-08', 'RoleName': 'helloworld-dev'}, 'url': 'https://iam.amazonaws.com/', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x7f8ea949c8d0>, 'has_streaming_input': False, 'auth_type': None}}
2018-07-29 13:46:52,841 - botocore.hooks - DEBUG - Event request-created.iam.GetRole: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8ea949c860>>
2018-07-29 13:46:52,842 - botocore.hooks - DEBUG - Event choose-signer.iam.GetRole: calling handler <function set_operation_specific_signer at 0x7f8ea98f5c80>
2018-07-29 13:46:52,842 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-07-29 13:46:52,842 - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:iam.amazonaws.com
x-amz-date:20180729T174652Z

content-type;host;x-amz-date
ca9448e942f5bbcd1b7b8fc14547e32585ca435311243df40a731c56086f5541
2018-07-29 13:46:52,842 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180729T174652Z
20180729/us-east-1/iam/aws4_request
3aa14d0d42403c3bf53c9fa67218a6e2a9c90f331d59606640fcdc3704cbd614
2018-07-29 13:46:52,842 - botocore.auth - DEBUG - Signature:
c890377e324df170e02d3bf1854f907e002b64c07a729607fca10f276d4204b3
2018-07-29 13:46:52,845 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2018-07-29 13:46:52,845 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): iam.amazonaws.com
2018-07-29 13:46:53,288 - botocore.endpoint - DEBUG - ConnectionError received when sending HTTP request.
Traceback (most recent call last):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
    conn.connect()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
    conn = self._new_conn()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
    conn.connect()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
    conn = self._new_conn()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/endpoint.py", line 222, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
2018-07-29 13:58:41,948 - botocore.hooks - DEBUG - Event needs-retry.iam.GetRole: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8be6784c88>
Traceback (most recent call last):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
    conn.connect()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
    conn = self._new_conn()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
    conn.connect()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
    conn = self._new_conn()
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/cli/__init__.py", line 447, in main
    return cli(obj={})
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/cli/__init__.py", line 183, in deploy
    deployed_values = d.deploy(config, chalice_stage_name=stage)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 330, in deploy
    return self._deploy(config, chalice_stage_name)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 341, in _deploy
    plan = self._plan_stage.execute(resources)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/deploy/planner.py", line 263, in execute
    result = handler(resource)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/deploy/planner.py", line 387, in _plan_managediamrole
    role_exists = self._remote_state.resource_exists(resource)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/deploy/planner.py", line 57, in resource_exists
    result = handler(resource)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/deploy/planner.py", line 95, in _resource_exists_managediamrole
    self._client.get_role_arn_for_name(resource.role_name)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/awsclient.py", line 369, in get_role_arn_for_name
    role = self.get_role(name)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/chalice/awsclient.py", line 376, in get_role
    role = client.get_role(RoleName=name)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/client.py", line 599, in _make_api_call
    operation_model, request_dict)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/endpoint.py", line 148, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/endpoint.py", line 177, in _send_request
    success_response, exception):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/endpoint.py", line 273, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/endpoint.py", line 222, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/jc/dev/chalice/chalice-demo/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Can anyone help me with this issue?

Answer 1

After some digging, I decided to change the proxy environments to:

http_proxy=http://127.0.0.1:3128
https_proxy=http://127.0.0.1:3128

And now the deploy works! I really do not know much about naming resolving, but, strangely enough, all other applications (including pip for example) work with http_proxy=http://localhost:3128 setting.