When do we need to add header to each audio frame

Question

I've done detaching audio (aac) from video (mp4). As I know, aac can play if and only each aac packet which is read from mp4 file is added ADTS header before writing to file.

I want to know when/ (which audio format) we need to add header ?

Answer 1

It depends on the codec/container combination. AAC (as you know) Always requires a header. In the .aac container, there is a ADTS header every frame. In .mp4 there is a single header Audio Specific Config in the mdhd box. MP3 on the other hand always has a header per frame no matter the container. The number of codec container combinations is enormous and impractical to list in the stack overflow format.