What is the first bytes in a Blob column SQlite Ad

Question

I have identified a random series of bytes inserted into any blob field when the database is manipulated via Adobe AIR. (from my results it appear to always start with bytes[12, ...] but I'm not sure of that)

I think it's a sizeinfo of the bytes, let me explain how I came to this conclusion.

First my context : I manipulate sqlite databases through Adobe AIR (client-side) and System.data.sqlite (C# server-side)

With System.data.sqlite if I read a Sqlite db filled with BLOB by Adobe AIR I have to get ride of those bytes appended in the beginning by AIR and then I have the binary data all well shaped. PERFECT!

With Adobe AIR if I tried to read a sqlite db filled with BLOB by System.data.Sqlite the data are corrupted I get an error! Obviously because I don't have the missing bytes researched by AIR.

Of course I tried to add those bytes by just copying a series of 3 bytes that I removed in my first case but then it returned the data partially and in the cases of images the last rows of pixels go all gray and in some images I have more or less grays lines. Because the data were corresponding to a series of images of the same ~4k size and I added 3 bytes from one of them and I had this result.

And Air will also sometimes throw this error :

Error: Error #2030: End of file was encountered.

So obviously those bytes give an info on the size but I don't really get how it does!?!

I tried to add a bytearray with 4k length it tends to add 3 bytes but I tried to add 4M and it goes up to 5 bytes.

I found this Question How do you convert 3 bytes into a 24 bit number in C#? and I thought that could be how the size info is stored.

But I still don't get it...

Answer 1

Thanks to FluorineFX (AMF for .NET) opensource project here is the answer.

Because in my Adobe AIR project I have to pass my air.ByteArray object as a parameter to store everything in the sqlite Blob field; AIR will serialize everything into AMF, a compact binary actionscript message format.

Page 11 Secion 3.14 ByteArray type

http://opensource.adobe.com/wiki/download/attachments/1114283/amf3_spec_05_05_08.pdf

The doc stipulate :

AMF 3 serializes this type using a variable length encoding 29-bit integer for the byte-length prefix followed by the raw bytes of the ByteArray.

But thats not all, I searched for an AMF .NET opensource project and founded FluorineFX. Looking into the code I identified that every AMF binaries is prefixed with a byte TypeCode which is 12 for a ByteArray:

   /// <summary>
   /// AMF ByteArray data type.
   /// </summary>
   public const byte ByteArray = 12;

Further search and again I found in the FluorineFX sources AMFReader.ReadAMF3ByteArray() and AMFWriter.WriteByteArray()

Which help me to quickly build what I need :

private static byte[] RemoveAMF3ByteArrayPrefixBytes(byte[] ar)
    {
        var ms = new MemoryStream(ar);
        var br = new BinaryReader(ms);

        // if first byte is AMF TypeCode for ByteArray
        if (br.Read() != 12)
            return ar;

        int handle = ReadAMF3IntegerData(br);
        bool inline = ((handle & 1) != 0);
        handle = handle >> 1;
        if (inline)
        {
            int length = handle;
            byte[] buffer = br.ReadBytes(length);
            return buffer;
        }

        return ar;
    }

    private static byte[] AddAMF3ByteArrayPrefixBytes(byte[] ar)
    {
        var ms = new MemoryStream();
        var bw = new BinaryWriter(ms);

        bw.Write((byte)12); // AMF TypeCode for ByteArray
        var handle = (int)ar.Length;
        handle = handle << 1;
        handle = handle | 1;
        WriteAMF3IntegerData(bw, handle);

        bw.Write(ar);

        return ms.ToArray();
    }

    /// <summary>
    /// Handle decoding of the variable-length representation which gives seven bits of value per serialized byte by using the high-order bit 
    /// of each byte as a continuation flag.
    /// </summary>
    /// <returns></returns>
    private static int ReadAMF3IntegerData(BinaryReader br)
    {
        int acc = br.ReadByte();
        if(acc < 128)
            return acc;
        else
        {
            acc = (acc & 0x7f) << 7;
            int tmp = br.ReadByte();
            if(tmp < 128)
                acc = acc | tmp;
            else
            {
                acc = (acc | tmp & 0x7f) << 7;
                tmp = br.ReadByte();
                if(tmp < 128)
                    acc = acc | tmp;
                else
                {
                    acc = (acc | tmp & 0x7f) << 8;
                    tmp = br.ReadByte();
                    acc = acc | tmp;
                }
            }
        }

        //To sign extend a value from some number of bits to a greater number of bits just copy the sign bit into all the additional bits in the new format.
        //convert/sign extend the 29bit two's complement number to 32 bit
        int mask = 1 << 28; // mask
        int r = -(acc & mask) | acc;
        return r;

        //The following variation is not portable, but on architectures that employ an 
        //arithmetic right-shift, maintaining the sign, it should be fast. 
        //s = 32 - 29;
        //r = (x << s) >> s;
    }

    private static void WriteAMF3IntegerData(BinaryWriter bw, int value)
    {
        //Sign contraction - the high order bit of the resulting value must match every bit removed from the number
        //Clear 3 bits 
        value &= 0x1fffffff;
        if (value < 0x80)
            bw.Write((byte)value);
        else
            if (value < 0x4000)
            {
                bw.Write((byte)(value >> 7 & 0x7f | 0x80));
                bw.Write((byte)(value & 0x7f));
            }
            else
                if (value < 0x200000)
                {
                    bw.Write((byte)(value >> 14 & 0x7f | 0x80));
                    bw.Write((byte)(value >> 7 & 0x7f | 0x80));
                    bw.Write((byte)(value & 0x7f));
                }
                else
                {
                    bw.Write((byte)(value >> 22 & 0x7f | 0x80));
                    bw.Write((byte)(value >> 15 & 0x7f | 0x80));
                    bw.Write((byte)(value >> 8 & 0x7f | 0x80));
                    bw.Write((byte)(value & 0xff));
                }
    }

I hope that will help someone else.

Answer 2

Thanks a lot, that helped me solve the problem. Here is how to do it with Java code:

Add org.granite granite-core dependency to your project

Init GDS with boilerplate code

GraniteConfig graniteConfig = new GraniteConfig(null, null, null, null);
ServicesConfig servicesConfig = new ServicesConfig(null, null, false);
Map<String, Object> applicationMap = new HashMap<String, Object>();
SimpleGraniteContext.createThreadIntance(graniteConfig, servicesConfig, applicationMap);

In my example, i read an image file that i will insert into a BLOB:

fis = new FileInputStream(file);
ps = connection.prepareStatement(INSERT_PICTURE);
ps.setString(1, key);
byte[] fileBytes = FileUtils.readFileToByteArray(file);
ByteArrayOutputStream out = new ByteArrayOutputStream();
AMF3Serializer ser = new AMF3Serializer(out);
ser.writeObject(fileBytes);
ser.flush();
ps.setBytes(2, out.toByteArray());

Works like a charm, thanks for the hint :)

Fabien