/bin/sh: line 62: to: command not found

Question

I have a python code in which I am calling a shell command. The part of the code where I did the shell command is:

try:
    def parse(text_list):
        text = '\n'.join(text_list)
        cwd = os.getcwd()
        os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
        synnet_output = subprocess.check_output(["echo '%s' | syntaxnet/demo.sh 2>/dev/null"%text], shell = True)
        os.chdir(cwd)
        return synnet_output
except Exception as e:
    sys.stdout.write(str(e))

Now, when i run this code on a local file with some sample input (I did cat /home/sree/example.json | python parse.py) it works fine and I get the required output. But I am trying to run the code with an input on my HDFS (the same cat command but input file path is from HDFS) which contains exactly the same type of json entries and it fails with an error:

/bin/sh: line 62: to: command not found
list index out of range

I read similar questions on Stack Overflow and the solution was to include a Shebang line for the shell script that is being called. I do have the shebang line #!/usr/bin/bash in demo.sh script.

Also, which bash gives /usr/bin/bash.

Someone please elaborate.

Answer 1

There was a problem with some special characters appearing in the text string that i was inputting to demo.sh. I solved this by storing text into a temporary file and sending the contents of that file to demo.sh.

That is:

try:
    def parse(text_list):
        text = '\n'.join(text_list)
        cwd = os.getcwd()
        with open('/tmp/data', 'w') as f:
            f.write(text)
        os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
        synnet_output = subprocess.check_output(["cat /tmp/data | syntaxnet/demo.sh 2>/dev/null"%text], shell = True)
        os.chdir(cwd)
        return synnet_output
except Exception as e:
    sys.stdout.write(str(e))

Answer 2

You rarely, if ever, want to combine passing a list argument with shell=True. Just pass the string:

synnet_output = subprocess.check_output("echo '%s' | syntaxnet/demo.sh 2>/dev/null"%(text,), shell=True)

However, you don't really need a shell pipeline here.

from subprocess import check_output
from StringIO import StringIO  # from io import StringIO in Python 3
synnet_output = check_output(["syntaxnet/demo.sh"],
                             stdin=StringIO(text),
                             stderr=os.devnull)