Understanding netty channel buffers and watermarks

I am trying to understand netty buffers and watermarks.

As a test case, I have a netty server which writes to a client, the client is blocked (essentially has a sleep of 10 seconds between each read)

Under normal I/O TCP sender would be throttled (sending is slowed down because of flow control) if the receiver is blocked, this is not the case here. The sender seems to keep writing and flushing data on each send. Where is this data being written? Is there going to be Flow control in the netty's flush() as well? See: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Flow_control
Is it is being written to an OS or TCP buffer, does netty channel have an internal buffer as well? If so how can I configure it?
I track bytesBeforeUnwritable but they do not seem to be decreasing
What is the default High and Low Watermark? I have not set anything in my application. Is there any way to use this instead?

Code below:

@Override
    public void channelRead(final ChannelHandlerContext ctx, Object msg) {

        if (server2OutboundChannel.isActive()) {
            if(server2OutboundChannel.isWritable()) {
                server2OutboundChannel.writeAndFlush(msg).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) {
                        if (future.isSuccess()) {
                            // was able to flush out data, start to read the next chunk
                            //System.out.println(server2OutboundChannel.bytesBeforeUnwritable());
                            ctx.channel().read();
                        } else {
                            future.channel().close();
                        }
                    }
                });
            }else{
                System.out.println("Channel is no longer writeable");
                System.out.println(server2OutboundChannel.bytesBeforeUnwritable());
                System.out.println(server2OutboundChannel.bytesBeforeWritable());
            }
        }
    }

Detailed steps to re-create with end-to-end source code:

Clone https://github.com/nipunarora/nettyDuplicator .
Specifically, the proxy code is here:

https://github.com/nipunarora/nettyDuplicator/tree/master/src/main/java/org/columbia/parikshan/proxy

Compile and Build:

mvn package
Start the server

sh scripts/Server.sh 3380
Start the netty proxy

sh scripts/nettyProxy.sh -l 0.0.0.0:3379 -o 127.0.0.1:3380
Start the client

sh scripts/Client.sh 127.0.0.1 3379
send "hello" as std input in client
netty blocks sending after some time and the bytesTillUnwritable do not decrease.

Where is this data being written? Is there going to be Flow control in the netty's flush() as well?

The data went to ChannelOutboundBuffer. There is no Flow control like tcp. The data will be kept in ChannelOutboundBuffer until there is space in tcp's send buffer.

Is it is being written to an OS or TCP buffer, does netty channel have an internal buffer as well? If so how can I configure it?

Netty has ChannelOutboundBuffer which keep data before send to OS'buffer. You can configure it like below.

    Bootstrap bootstrap = new Bootstrap();
    bootstrap.option(ChannelOption.SO_RCVBUF, soRcvBufSize);
    bootstrap.option(ChannelOption.SO_SNDBUF, soSndBufSize);
    bootstrap.option(ChannelOption.WRITE_BUFFER_HIGH_WATER_MARK, writeBufferHighWaterMark);

I track bytesBeforeUnwritable but they do not seem to be decreasing

I write a sample code that server write to a blocked client

Your proxy's AUTO_READ is false. NettyProxyFrontendHandler#channelRead will only be called when ctx.channel().read() (in the future listener) is called. The listener will be called after writeAndFlush is completed. writeAndFlush will generate a task and the task will is done when the msg is write to OS's buffer. If OS's buffer is filled, the task will be blocked. There is no chance that netty's buffer will become unwritable, it is always writeable.

What is the default High and Low Watermark? I have not set anything in my application. Is there any way to use this instead?

You can check the defualt water mark in DefaultChannelConfig -> WriteBufferWaterMark.DEFAULT. When data in ChannelOutboundBuffer > high water mark the isWritable return false, < low water mark return true.

 /**
 * Returns {@code true} if and only if the I/O thread will perform the
 * requested write operation immediately.  Any write requests made when
 * this method returns {@code false} are queued until the I/O thread is
 * ready to process the queued write requests.
 */
boolean isWritable();