I have a golang program that uses unmarshall from std "encoding/json" package keeps increasing in size (memory leak). A diagram of memory profile using pprof shows memory increasing at json (* decodeState) objectInterface. I want to understand how and why it could be happening to fix the issue.
I have tried several things at upper level like release of returned value to avoid the leak, but with no success.
func (j JSONEncoding) From(b []byte, msg interface{}) (interface{}, error) {
err := json.Unmarshal(b, &msg)
return msg, err
}
pprof top5 shows this call, and the details below;
4096.89kB 24.43% 24.43% 4096.89kB 24.43% encoding/json.(*decodeState).objectInterface
(pprof) list objectInterface
Total: 20.65MB
ROUTINE ======================== encoding/json.(*decodeState).objectInterface in /usr/local/go/src/encoding/json/decode.go
6MB 7.50MB (flat, cum) 36.32% of Total
. . 1061: return v
. . 1062:}
. . 1063:
. . 1064:// objectInterface is like object but returns map[string]interface{}.
. . 1065:func (d *decodeState) objectInterface() map[string]interface{} {
3MB 3MB 1066: m := make(map[string]interface{})
. . 1067: for {
. . 1068: // Read opening " of string key or closing }.
.
. deleted code
.
. . 1095:
. . 1096: // Read value.
3MB 4.50MB 1097: m[key] = d.valueInterface()
using topN, and using visualization I can see this box increasing with time/processing.
This unmarshall is called in a loop but nothing is saved that could be a reason for the leak. I am not sure how and what of something to be done so that this leak is avoided.
Update: The memory leak was more of a memory accrual, in the code at some other place. While trying to write a minimum code that shows the problem, I was not able to reproduce, and had to dig all the code to find out that an internal library was using a map to cache and the cache cleaner was not working properly.
My problem was that pprof is giving info on who is allocating, but not where it is being kept. I think my question should have been, how can we find out which objects have references of what size. I know that one allocated reference could be in multiple places but such information would easily help in finding this kind of issue.