curl: (3) Illegal characters found in URL

Question

I want to bulk lookup ip details at ipinfo.io Here is my code. $ cat ips.txt | xargs -I% curl http://ipinfo.io/%/region

The file "ips.txt" contains three ip addresses each on a separate line: (1) 8.8.8.8 (2) 8.8.4.4 (3) 1.2.3.4

This resolves only the last ip address. It should give (1) California (2) Colorado (3) Washington. I get the below:

curl: (3) Illegal characters found in URL curl: (3) Illegal characters found in URL Washington

If I write ips.txt with only one ip address (for example 8.8.8.8) I get good results. I think there is something wrong with either my text file or the way I am using cat. Can you help me clean my code so that all three ip addresses are resolved?

Per request, here are details of my setup.

$ uname -a
CYGWIN_NT-10.0 OFFICECOMP 2.3.1(0.291/5/3) 2015-11-14 12:44 x86_64 Cygwin
$ curl -V
curl 7.45.0 (x86_64-unknown-cygwin) libcurl/7.45.0 OpenSSL/1.0.2d zlib/1.2.8 lib idn/1.29 libssh2/1.5.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: Debug IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets Metali

Answer 1

So to summarize, as a step-by step for future reference,

Running uname -A can reveal what system is being used for example revealing here it is Cygwin.

This leads us to thinking about Windows vs Linux differences.

One of the known differences are windows/dos line endings. This can be revealed, as well as with generally checking all contents of a text file, with:

cat -A ips.txt

Now if you find it contains ^M line endings, as it did here, it means the line endings are dos/windows and not *nix line endings (which would show only lines ending with $)

To fix this, simply run

dos2unix ips.txt

Now, if you run your original command with this fixed input file, it seems CURL is happy with it and it works.

Thank you as well MistaGill, I learned about ipinfo.io from this post.

Answer 2

I ran into a similar situation, and a possible quick fix is to add a -d "\n" flag to xargs:

cat ips.txt | xargs -d "\n" -I% curl http://ipinfo.io/%/region

This prevents new lines from being interpreted as part of the argument being passed to curl.