I wonder if it is possible in Flink to share the state among operators.
Say, for instance, that I have partitioning by key on an operator and I need a piece of state of partition A
inside partition C
(for any reason) (fig 1.a), or I need the state of operator C
in downstream operator F
(fig 1.b).
I know it is possible to broadcast
records to all partitions. So, if you include the internal state of an operator inside the records, you can share your internal state with downstream operators.
However, this could be an expensive operation instead of simply letting op1
specifically ask for op2
state.
Are the recent developments around queryable state moving towards this concept or they are meant only to let an external user query the internal state of the topology?
Thank you in advance for your insights