bash tab completion with spaces

Question

I'm having a problem with bash-completion when the possible options may contain spaces.

Let's say I want a function which echoes the first argument:

function test1() {
        echo $1
}

I generate a list of possible completion options (some have spaces, some not), but I don't manage to handle spaces correctly.

function pink() {
    # my real-world example generates a similar string using awk and other commands
    echo "nick\ mason syd-barrett david_gilmour roger\ waters richard\ wright"
}

function _test() {
    cur=${COMP_WORDS[COMP_CWORD]}
    use=`pink`
    COMPREPLY=( $( compgen -W "$use" -- $cur ) )
}
complete -o filenames -F _test test

When I try this, I get:

$ test <tab><tab>
david_gilmour  nick           roger          waters
mason          richard        syd-barrett    wright
$ test r<tab><tab>
richard  roger    waters   wright

which is obviously not what I meant.

If I don't assign an array to COMPREPLY, i.e. only $( compgen -W "$use" -- $cur ), I get it working if only one option remains:

$ test n<tab>
$ test nick\ mason <cursor>

But if several options remain, they are all printed within single quotes:

$ test r<tab><tab>
$ test 'roger waters
richard wright' <cursor>

There must be something wrong with my COMPREPLY variable, but I can't figure out what...

(running bash on solaris, in case that makes a difference...)

Answer 1

Custom tab-completing words which might include whitespace is annoyingly difficult. And as far as I know there is no elegant solution. Perhaps some future version of compgen will be kind enough to produce an array rather than outputting possibilities one line at a time, and even accept the wordlist argument from an array. But until then, the following approach may help.

It's important to understand the problem, which is that ( $(compgen ... ) ) is an array produced by splitting the output of the compgen command at the characters in $IFS, which by default is any whitespace character. So if compgen returns:

roger waters
richard wright

then COMPREPLY will effectively be set to the array (roger waters richard wright), for a total of four possible completions. If you instead use ( "$(compgen ...)"), then COMPREPLY will be set to the array ($'roger waters\nrichard wright'), which has only one possible completion (with a newline inside the completion). Neither of those are what you want.

If none of the possible completions has a newline character, then you could arrange for the compgen return to be split at the newline character by temporarily resetting IFS and then restoring it. But I think a more elegant solution is to just use mapfile:

_test () { 
    cur=${COMP_WORDS[COMP_CWORD]};
    use=`pink`;
    ## See note at end of answer w.r.t. "$cur" ##
    mapfile -t COMPREPLY < <( compgen -W "$use" -- "$cur" )
}

The mapfile command places the lines sent by compgen to stdout into the array COMPREPLY. (The -t option causes the trailing newline to be removed from each line, which is almost always what you want when you use mapfile. See help mapfile for more options.)

This doesn't deal with the other annoying part of the problem, which is mangling the wordlist into a form acceptable by compgen. Since compgen does not allow multiple -W options, and nor does it accept an array, the only option is to format a string in a such a way that bash word-splitting (with quotes and all) would generate the desired list. In effect, that means manually adding escapes, as you did in your function pink:

pink() {
    echo "nick\ mason syd-barrett david_gilmour roger\ waters richard\ wright"
}

But that's accident-prone and annoying. A nicer solution would allow the specification of the alternatives directly, particularly if the alternatives are being generated in some fashion. A good way of generating alternatives which might include whitespace is to put them into an array. Given an array, you can make good use of printf's %q format to produce a properly-quoted input string for compgen -W:

# This is a proxy for a database query or some such which produces the alternatives
cat >/tmp/pink <<EOP
nick mason
syd-barrett
david_gilmour
roger waters
richard wright
EOP

# Generate an array with the alternatives
mapfile -t pink </tmp/pink

# Use printf to turn the array into a quoted string:
_test () { 
    mapfile -t COMPREPLY < <( compgen -W "$(printf '%q ' "${pink[@]}")" -- "$2" )
}

As written, that completion function does not output completions in a form which will be accepted by bash as single words. In other words, the completion roger waters is generated as roger waters instead of roger\ waters. In the (likely) case that the goal is to produce correctly quoted completions, it is necessary to add escapes a second time, after compgen filters the completion list:

_test () {
    declare -a completions
    mapfile -t completions < <( compgen -W "$(printf '%q ' "${pink[@]}")" -- "$2" )
    local comp
    COMPREPLY=()
    for comp in "${completions[@]}"; do
        COMPREPLY+=("$(printf "%q" "$comp")")
    done
}

Note: I replaced the computation of $cur with $2, since the function invoked through complete -F is passed the command as $1 and the word being completed as $2. (It's also passed the previous word as $3.) Also, it's important to quote it, so that it doesn't get word-split on its way into compgen.

Answer 2

Okay, this crazy contraption draws heavily on rici’s solution, and not only fully works, but also quotes any completions that need it, and only those.

pink() {
    # simulating actual awk output
    echo "nick mason"
    echo "syd-barrett"
    echo "david_gilmour"
    echo "roger waters"
    echo "richard wright"
}

_test() {
  cur=${COMP_WORDS[COMP_CWORD]}
  mapfile -t patterns < <( pink )
  mapfile -t COMPREPLY < <( compgen -W "$( printf '%q ' "${patterns[@]}" )" -- "$cur" | awk '/ / { print "\""$0"\"" } /^[^ ]+$/ { print $0 }' )
}

complete -F _test test

So as far as I could test it, it fully implements ls-like behavior, minus the path-specific parts.

Verbose example

Here’s a more verbose version of the _test function, so it becomes a bit more understandable:

_test() {
  local cur escapedPatterns
  cur=${COMP_WORDS[COMP_CWORD]}
  mapfile -t patterns < <( pink )
  escapedPatterns="$( printf '%q ' "${patterns[@]}" )"
  mapfile -t COMPREPLY < <( compgen -W "$escapedPatterns" -- "$cur" | quoteIfNeeded )
}

quoteIfNeeded() {
  # Only if it contains spaces. Otherwise return as-is.
  awk '/ / { print "\""$0"\"" } /^[^ ]+$/ { print $0 }'
}

None of this is even remotely optimized for efficiency. Then again, this is only tab completion, and it’s not causing a noticeable delay for any reasonably large list of completions.

It works by:

1. Pulling the awk output into an array, using mapfile.
2. Escaping the array and putting it into a string.
3. Having a single space behind the %q as a separation marker.
4. Quoting $cur, Very important!
5. Quoting the output of compgen. And only if it contains spaces.
6. Feeding that output into COMPREPLY, using another mapfile call.
7. Not using -o filenames.

And it only works with all those tricks. It fails if even a single one is missing. Trust me; I’ve tried. ;)

Answer 3

If you need to process the data from the string you can use Bash's built-in string replacement operator.

function _test() {
    local iter use cur
    cur=${COMP_WORDS[COMP_CWORD]}
    use="nick\ mason syd-barrett david_gilmour roger\ waters richard\ wright"
    # swap out escaped spaces temporarily
    use="${use//\\ /___}"
    # split on all spaces
    for iter in $use; do
        # only reply with completions
        if [[ $iter =~ ^$cur ]]; then
            # swap back our escaped spaces
            COMPREPLY+=( "${iter//___/ }" )
        fi
    done
}