LengthFieldBasedFrameDecoder not parsing correctly

Question

I am unit testing a netty pipeline using the frame based decoder. It looks like the framing is incorrect if I use buffer size that is smaller that the largest frame. I am testing with a file that contains two messages. The length field is the second work and includes the length of the entire message including the length field and the work before it.

 new LengthFieldBasedFrameDecoder(65536, 4, 4, -8, 0)

I am reading a file with various block sizes. The size of the first message is 348 bytes, the second is 456 bytes. If block size of 512, 3456, or larger, is used both messages are read and correctly framed to the next handler which for diagnostic purposes will print out as a hexadecimal string the contents of the buffer it received. If a smaller block size is used framing errors occur. The code used to read and write the file is shown below.

public class NCCTBinAToCSV {
    private static String inputFileName = "/tmp/combined.bin";
    private static final int BLOCKSIZE = 456;
    public static void main(String[] args) throws Exception {
        byte[] bytes = new byte[BLOCKSIZE];
        EmbeddedChannel channel = new EmbeddedChannel(
                new LengthFieldBasedFrameDecoder(65536, 4, 4, -8, 0),
                new NCCTMessageDecoder(),
                new StringOutputHandler());
        FileInputStream fis = new FileInputStream(new File(inputFileName));
        int bytesRead = 0;
        while ((bytesRead = fis.read(bytes)) != -1) {
            ByteBuf buf = Unpooled.wrappedBuffer(bytes, 0, bytesRead);
            channel.writeInbound(buf);
        }
        channel.flush();
    }
}

Output from a successful run with block size of 356 bytes is show below (with the body of the messages truncated for brevity

LOG:DEBUG 2017-04-24 04:19:24,675[main](netty.NCCTMessageDecoder) -  com.ticomgeo.mtr.ncct.netty.NCCTMessageDecoder.decode(NCCTMessageDecoder.java:21) ]received 348 bytes

Frame Start========================================

(byte) 0xbb,   (byte) 0x55,   (byte) 0x05,   (byte) 0x16,   
(byte) 0x00,   (byte) 0x00,   (byte) 0x01,   (byte) 0x5c,   
(byte) 0x01,   (byte) 0x01,   (byte) 0x02,   (byte) 0x02,   
(byte) 0x05,   (byte) 0x00,   (byte) 0x00,   (byte) 0x00,   
(byte) 0x50,   (byte) 0x3a,   (byte) 0xc9,   (byte) 0x17,

....   
Frame End========================================

Frame Start========================================

(byte) 0xbb,   (byte) 0x55,   (byte) 0x05,   (byte) 0x1c,   
(byte) 0x00,   (byte) 0x00,   (byte) 0x01,   (byte) 0xc8,   
(byte) 0x01,   (byte) 0x01,   (byte) 0x02,   (byte) 0x02,   
(byte) 0x05,   (byte) 0x00,   (byte) 0x00,   (byte) 0x00,   
(byte) 0x04,   (byte) 0x02,   (byte) 0x00,   (byte) 0x01,  

If I change the block size to 256, the wrong bytes seem to be read as the length field.

Exception in thread "main" io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 65536: 4294967040 - discarded
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:499)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:477)
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:403)

Answer 1

TL;DR; Your problem is caused because netty reuses the passed in bytebuf, and then you are overwriting the contents.

LengthFieldBasedFrameDecoder is designed through inheritance to reuse the passed in ByteBuf, because it is useless to let the object decay through garbage collection when you can reuse it because its reference count is 1. The problem however comes from the fact that you are changing the internals of the passed in bytebuf, and therefore changing the frame on the fly. Instead of making a wrappedBuffer, that uses your passed in variable as storage, you should use copiedBuffer, because that one properly makes a copy of it, so the internals of LengthFieldBasedFrameDecoder can do freely things with it.