How to urlencode data for curl command?

Question

I am trying to write a bash script for testing that takes a parameter and sends it through curl to web site. I need to url encode the value to make sure that special characters are processed properly. What is the best way to do this?

Here is my basic script so far:

#!/bin/bash
host=${1:?'bad host'}
value=$2
shift
shift
curl -v -d "param=${value}" http://${host}/somepath $@

Answer 1

You can emulate javascript's encodeURIComponent in perl. Here's the command:

perl -pe 's/([^a-zA-Z0-9_.!~*()'\''-])/sprintf("%%%02X", ord($1))/ge'

You could set this as a bash alias in .bash_profile:

alias encodeURIComponent='perl -pe '\''s/([^a-zA-Z0-9_.!~*()'\''\'\'''\''-])/sprintf("%%%02X",ord($1))/ge'\'

Now you can pipe into encodeURIComponent:

$ echo -n 'hèllo wôrld!' | encodeURIComponent
h%C3%A8llo%20w%C3%B4rld!

Answer 2

Here's a one-line conversion using Lua, similar to blueyed's answer except with all the RFC 3986 Unreserved Characters left unencoded (like this answer):

url=$(echo 'print((arg[1]:gsub("([^%w%-%.%_%~])",function(c)return("%%%02X"):format(c:byte())end)))' | lua - "$1")

Additionally, you may need to ensure that newlines in your string are converted from LF to CRLF, in which case you can insert a gsub("\r?\n", "\r\n") in the chain before the percent-encoding.

Here's a variant that, in the non-standard style of application/x-www-form-urlencoded, does that newline normalization, as well as encoding spaces as '+' instead of '%20' (which could probably be added to the Perl snippet using a similar technique).

url=$(echo 'print((arg[1]:gsub("\r?\n", "\r\n"):gsub("([^%w%-%.%_%~ ]))",function(c)return("%%%02X"):format(c:byte())end):gsub(" ","+"))' | lua - "$1")

Answer 3

The following is based on Orwellophile's answer, but solves the multibyte bug mentioned in the comments by setting LC_ALL=C (a trick from vte.sh). I've written it in the form of function suitable PROMPT_COMMAND, because that's how I use it.

print_path_url() {
  local LC_ALL=C
  local string="$PWD"
  local strlen=${#string}
  local encoded=""
  local pos c o

  for (( pos=0 ; pos<strlen ; pos++ )); do
     c=${string:$pos:1}
     case "$c" in
        [-_.~a-zA-Z0-9/] ) o="${c}" ;;
        * )               printf -v o '%%%02x' "'$c"
     esac
     encoded+="${o}"
  done
  printf "\033]7;file://%s%s\007" "${HOSTNAME:-}" "${encoded}"
}

Answer 4

Ruby, for completeness

value="$(ruby -r cgi -e 'puts CGI.escape(ARGV[0])' "$2")"

Answer 5

Here is my version for busybox ash shell for an embedded system, I originally adopted Orwellophile's variant:

urlencode()
{
    local S="${1}"
    local encoded=""
    local ch
    local o
    for i in $(seq 0 $((${#S} - 1)) )
    do
        ch=${S:$i:1}
        case "${ch}" in
            [-_.~a-zA-Z0-9]) 
                o="${ch}"
                ;;
            *) 
                o=$(printf '%%%02x' "'$ch")                
                ;;
        esac
        encoded="${encoded}${o}"
    done
    echo ${encoded}
}

urldecode() 
{
    # urldecode <string>
    local url_encoded="${1//+/ }"
    printf '%b' "${url_encoded//%/\\x}"
}

Answer 6

Another option is to use jq:

jq -s -R -r @uri

-s (--slurp) reads input lines into an array and -s -R (--slurp --raw-input) reads the input into a single string. -r (--raw-output) outputs the contents of strings instead of JSON string literals.

Or this percent-encodes all bytes:

xxd -p|tr -d \\n|sed 's/../%&/g'