modify a datastream on the fly

Question

I need to hijack and modify a datastream. The stream consists of fixed-width commands. Each command is a new line, and the documentation says that each command starts and ends with an STX/ETX pair (start and end of text)

The sending system is using serial, but is attacked to an iPocket device that communicates over IP to our PBX. From what I can tell it's just converting the serial to telnet, which should keep things pretty simple hopefully.

The string I need to look for starts with NAM. The structure is something like this:

STX NAM EXT# LASTNAME,FIRSTNAME ETX

I need to replace ,FIRSTNAME with whitespace so it doesn't change the length of the command.

I've been attempting the following, and while it passes the data end to end in both directions as required, and removes the needed data, it isn't maintaining the length of the command.

ipocket <-> nc -kl 1100 | sed 's/,[^,]*/ /g' | nc target_ip target_port <-> PBX

I'll be using a linux box for this task. I'm reasonably certain that this could be done quite simply in perl or python but I haven't got a clue where to start. Any assistance would be greatly appreciated!

Answer 1

In the Perl, with added check that the string starts with NAM and replacing really the ,FIRSTNAME:

nc -kl 1100 | \
perl -pe '/^STX NAM / && do { s/(,FIRSTNAME)/" " x length("$1")/ge }' | \
nc <...>

In Perl s///e as expected does substitution, but evaluates the replacement string as an expression. Operator x makes a new string, duplicating string on the left number of times given on the right. Non-matching strings are obviously not modified.

Answer 2

Yes, the iPocket does raw pass-through of the data. The options are typically for packetization of the incoming data which won't actually affect what you're trying to do here. (I wrote the firmware for that device.)

However, none of the command-line answers here will work because they are line-based and your data is not. That is to say, there are to linefeeds in the data stream that sed and perl use as "packet" boundaries.

I don't believe there is any way you're going to be able to accomplish this from the command-line. You're going to have to write a simple program that reads an incoming TCP stream, looks for the STX/ETX framing, replaces the characters as you wish, and then writes the data out the other side. (Don't forget pass-through the back direction, too.)

Python is probably the simplest way to do this.

Answer 3

Here's a sed version:

sed 'h;s/[^,]*,\([^ ]*\) ETX/\1/;s/./ /g;x;s/,.*/,/;G;s/\n//;s/$/ ETX/'

I'd be willing to bet that not only is the command fixed width, but that the fields are also fixed width. If that's the case then something like this would probably work:

sed 's/\(.\{22\}\).\{9\}\(.\{4\}\)/\1         \2/'

or

sed -r 's/(.{22}).{9}(.{4})/\1         \2/'

or

sed -r 's/STX (.{18}).{9} ETX/STX \1          ETX/'

Answer 4

Instead of the sed command, you can use this:

perl -pe 's/(,\w+)/" " x length $1/ge'

The e option on the substitution means that the right side of the s/// is evaluated as a Perl expression. In this case, the expression returns the correct number of spaces, based on the length of the captured match.

Answer 5

This should do it, its a one liner

perl -pe '/^STX\s\w+\s\d+#\s\w+,(\w+)\sETX$/;$len=length($1);s/$1/" " x $len/e'

I tested using.

echo "STX NAM 100# LASTNAME,FIRSTNAME ETX" | perl -pe '/^STX\s\w+\s\d+#\s\w+,(\w+)\sETX$/;$len=length($1);s/$1/" " x $len/e'

and it returns

STX NAM 100# LASTNAME,          ETX

To ensure the strings are the same length i tested.

echo "STX NAM 100# LASTNAME,FIRSTNAME ETX" | perl -pe '/^STX\s\w+\s\d+#\s\w+,(\w+)\sETX$/;$len=length($1);s/$1/" " x $len/e'| perl -pe 'print length($_);'

Gives 36.

echo "STX NAM 100# LASTNAME,FIRSTNAME ETX"|perl -pe 'print length($_);'

Gives 36.