How can I append to an existing java.io.ObjectStre

Question

This question already has an answer here:

• Appending to an ObjectOutputStream 5 answers

As for now I will get java.io.StreamCorruptedException when I try to append an Object. I have searched the Internet for a way to overcome that. The answer I found so far is it can't be done. A way around this problem is to write the objects into a list and then write the list to the file.

But I have to overwrite that file everytime when I add new objects. It seems not to be the optimal solution in overtime.

Is there a way to append objects to an existing object stream?

Answer 1

The best article I've found on this topic is: http://codify.flansite.com/2009/11/java-serialization-appending-objects-to-an-existing-file/

The "solution" that overrides ObjectOutputStream is simply wrong. I've just finished investigating a bug that was caused by that (wasting two precious days). Not only that it would sometimes corrupt the serialized file but even managed to read without throwing exceptions and in the end providing garbage data (mixing fields). For those in disbelief, I'm attaching some code that exposes the problem:

import java.io.*;
import java.util.HashMap;
import java.util.Map;

public class Main {

    public static void main(String[] args) throws Exception {

        File storageFile = new File("test");
        storageFile.delete();

        write(storageFile, getO1());
        write(storageFile, getO2());
        write(storageFile, getO2());

        ObjectInputStream ois = new ObjectInputStream(new FileInputStream(storageFile));
        read(ois, getO1());
        read(ois, getO2());
        read(ois, getO2());
    }

    private static void write(File storageFile, Map<String, String> o) throws IOException {
        ObjectOutputStream oos = getOOS(storageFile);
        oos.writeObject(o);
        oos.close();
    }

    private static void read(ObjectInputStream ois, Map<String, String> expected) throws ClassNotFoundException, IOException {
        Object actual = ois.readObject();
        assertEquals(expected, actual);
    }

    private static void assertEquals(Object o1, Object o2) {
        if (!o1.equals(o2)) {
            throw new AssertionError("\n expected: " + o1 + "\n actual:   " + o2);
        }
    }

    private static Map<String, String> getO1() {
        Map<String, String> nvps = new HashMap<String, String>();
        nvps.put("timestamp", "1326382770000");
        nvps.put("length", "246");
        return nvps;
    }

    private static Map<String, String> getO2() {
        Map<String, String> nvps = new HashMap<String, String>();
        nvps.put("timestamp", "0");
        nvps.put("length", "0");
        return nvps;
    }

    private static ObjectOutputStream getOOS(File storageFile) throws IOException {
        if (storageFile.exists()) {
            // this is a workaround so that we can append objects to an existing file
            return new AppendableObjectOutputStream(new FileOutputStream(storageFile, true));
        } else {
            return new ObjectOutputStream(new FileOutputStream(storageFile));
        }
    }

    private static class AppendableObjectOutputStream extends ObjectOutputStream {

        public AppendableObjectOutputStream(OutputStream out) throws IOException {
            super(out);
        }

        @Override
        protected void writeStreamHeader() throws IOException {
            // do not write a header
        }
    }
}

As stated in that article, you can use one of the following solutions:

Solution #1: Fake Multiple file in a Single Stream

...

Write your “transaction” to a ByteArrayOutputStream, then write the length and contents of this ByteArrayOutputStream to a file via the DataOutputStream.

Solution #2: Reopen and Skip

Another solution involves saving the file position using:

long pos = fis.getChannel().position();

closing the file, reopening the file, and skipping to this position before reading the next transaction.

Answer 2

Many thanks to George Hategan for the problem exposing code. I examined it for a while too. Then, it hit me. If you're using a sub-classed ObjectOutputStream with an override of the writeStreamHeader() method to write data, you must use the parallel sub-classed ObjectInputStream with an override of the readStreamHeader() method to read the data. Of course, we can zig-zag between different implementations of writing and reading objects, but as long as we use the corresponding pairs of sub-classes in the write/read process - we'll be (hopefully) fine. Tom.

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;

public class SerializationDemo {

    public static void main(String[] args) throws Exception {
        File storageFile = new File("test.ser");
        storageFile.delete();
        write(storageFile, getO1());
        write(storageFile, getO2());
        write(storageFile, getO2());
        FileInputStream fis = new FileInputStream(storageFile);
        read(fis, getO1());
        read(fis, getO2());
        read(fis, getO2());
        fis.close();
    }

    private static void write(File storageFile, Map<String, String> o)
                    throws IOException {
        ObjectOutputStream oos = getOOS(storageFile);
        oos.writeObject(o);
        oos.flush();
        oos.close();
    }

    private static void read(FileInputStream fis, Map<String, String> expected)
                    throws ClassNotFoundException, IOException {
        Object actual = getOIS(fis).readObject();
        assertEquals(expected, actual);
        System.out.println("read serialized " + actual);
    }

    private static void assertEquals(Object o1, Object o2) {
        if (!o1.equals(o2)) {
            throw new AssertionError("\n expected: " + o1 + "\n actual:   " + o2);
        }
    }

    private static Map<String, String> getO1() {
        Map<String, String> nvps = new HashMap<String, String>();
        nvps.put("timestamp", "1326382770000");
        nvps.put("length", "246");
        return nvps;
    }

    private static Map<String, String> getO2() {
        Map<String, String> nvps = new HashMap<String, String>();
        nvps.put("timestamp", "0");
        nvps.put("length", "0");
        return nvps;
    }

    private static ObjectOutputStream getOOS(File storageFile)
                    throws IOException {
        if (storageFile.exists()) {
            // this is a workaround so that we can append objects to an existing file
            return new AppendableObjectOutputStream(new FileOutputStream(storageFile, true));
        } else {
            return new ObjectOutputStream(new FileOutputStream(storageFile));
        }
    }

    private static ObjectInputStream getOIS(FileInputStream fis)
                    throws IOException {
        long pos = fis.getChannel().position();
        return pos == 0 ? new ObjectInputStream(fis) : 
            new AppendableObjectInputStream(fis);
    }

    private static class AppendableObjectOutputStream extends
                    ObjectOutputStream {

        public AppendableObjectOutputStream(OutputStream out)
                        throws IOException {
            super(out);
        }

        @Override
        protected void writeStreamHeader() throws IOException {
            // do not write a header
        }
    }

    private static class AppendableObjectInputStream extends ObjectInputStream {

        public AppendableObjectInputStream(InputStream in) throws IOException {
            super(in);
        }

        @Override
        protected void readStreamHeader() throws IOException {
            // do not read a header
        }
    }
}

Answer 3

It is actually pretty easy to do. When you are adding to an existing stream you need to use a subclass of ObjectOutStream that overrides writeStreamHeader so that a second header is not written in the middle of the file. For example

class NoHeaderObjectOutputStream extends ObjectOutputStream {
  public NoHeaderObjectOutputStream(OutputStream os) {
    super(os);
  }
  protected void writeStreamHeader() {}
}

Then just use a standard ObjectInputStream to read the whole file.

Answer 4

You would need to create a new ObjectInputStream to match every ObjectOutputStream. I don't know a way to transfer state from a complete ObjectInputStream to an ObjectOutputStream (without a complete reimplementation, which is a bit tricky in pure Java anyway).