How to save OpenCV Keypoint features to database?

Question

My project is recognition the leaf on Android using OpenCV library. I am using ORB detection to get the keypoint of image and use ORB descriptor to get the feature of the keypoint. This is the code that i use:

bmp=BitmapFactory.decodeResource(getResources(),R.drawable.t1);
Utils.bitmapToMat(bmp, mat);
FeatureDetector detector = FeatureDetector.create(FeatureDetector.ORB);
detector.detect(mat, keypoints);
DescriptorExtractor extractor = DescriptorExtractor.create(DescriptorExtractor.ORB);
extractor.compute(mat, keypoints, features);

Source : http://answers.opencv.org/question/6260/orb-features/

But every i input the same image, the keypoint of that image always different. Can i save the feature of keypoint to database if that always different? Or should i save the image to save the feature data? If that can save to database, how can i do that??

Answer 1

In my opinion the most universal way to store the keypoints is to first convert them to a data-interchange format like JSON.

After you are able to do that conversion you have a lot of flexibility to store it. JSON is easily converted to a String and/or sent through a network connection.

With OpenCV C++ you are able to store data as YAML, but that is not available for Android yet.

To parse JSON in Java you can use this easy to use library Google GSON.

And here is my first attempt to do exactly that:

 public static String keypointsToJson(MatOfKeyPoint mat){
    if(mat!=null && !mat.empty()){          
        Gson gson = new Gson();

        JsonArray jsonArr = new JsonArray();            

        KeyPoint[] array = mat.toArray();
        for(int i=0; i<array.length; i++){
            KeyPoint kp = array[i];

            JsonObject obj = new JsonObject();

            obj.addProperty("class_id", kp.class_id); 
            obj.addProperty("x",        kp.pt.x);
            obj.addProperty("y",        kp.pt.y);
            obj.addProperty("size",     kp.size);
            obj.addProperty("angle",    kp.angle);                          
            obj.addProperty("octave",   kp.octave);
            obj.addProperty("response", kp.response);

            jsonArr.add(obj);               
        }

        String json = gson.toJson(jsonArr);         

        return json;
    }
    return "{}";
}

public static MatOfKeyPoint keypointsFromJson(String json){
    MatOfKeyPoint result = new MatOfKeyPoint();

    JsonParser parser = new JsonParser();
    JsonArray jsonArr = parser.parse(json).getAsJsonArray();        

    int size = jsonArr.size();

    KeyPoint[] kpArray = new KeyPoint[size];

    for(int i=0; i<size; i++){
        KeyPoint kp = new KeyPoint(); 

        JsonObject obj = (JsonObject) jsonArr.get(i);

        Point point = new Point( 
                obj.get("x").getAsDouble(), 
                obj.get("y").getAsDouble() 
        );          

        kp.pt       = point;
        kp.class_id = obj.get("class_id").getAsInt();
        kp.size     =     obj.get("size").getAsFloat();
        kp.angle    =    obj.get("angle").getAsFloat();
        kp.octave   =   obj.get("octave").getAsInt();
        kp.response = obj.get("response").getAsFloat();

        kpArray[i] = kp;
    }

    result.fromArray(kpArray);

    return result;
}

Answer 2

Although I have not run timings, I suspect that going through formats such as XML or JSON would require more space and computing resources than using primitives.

Below is some code that can be used to save and retrieve MatOfKeyPoint to SQLite database in an Android environment. This is using OpenCV 2.4.11 so SIFT is available.

What you see when you run this application is your test image (which you need to supply and put in the drawable folder) with added keypoints.

The siftTest() method starts by computing keyPoints which is MatOfKeyPoint type. The code saves the underlying data of that object in the database, then reads that data out and creates a new object keyPointsFromDb, the contents of which are applied to the original image and the result is displayed.

public class MainActivity extends AppCompatActivity {
    static {
        System.loadLibrary("native-lib");
        System.loadLibrary("opencv_java");
        System.loadLibrary("nonfree");

    }
    private ImageView imageView;
    private Bitmap inputImage; // make bitmap from image resource
    private FeatureDetector detector = FeatureDetector.create(FeatureDetector.SIFT);

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        inputImage = BitmapFactory.decodeResource(getResources(), R.drawable.test);
        imageView = (ImageView) this.findViewById(R.id.imageView);
        siftTest();
    }

    public void siftTest() {
        Mat rgba = new Mat();
        Utils.bitmapToMat(inputImage, rgba);
        MatOfKeyPoint keyPoints = new MatOfKeyPoint();
        Imgproc.cvtColor(rgba, rgba, Imgproc.COLOR_RGBA2GRAY);
        detector.detect(rgba, keyPoints);

        // Save to database
        MatchingDatabaseAdapter.addKeypoints(keyPoints, getApplicationContext());

        // Opens database cursor
        MatchingDatabaseAdapter.getAllRecordsCursor(getApplicationContext()); 

        // Gets the first item in the database (as an example... you could loop through many/all)
        MatOfKeyPoint keyPointsFromDb = MatchingDatabaseAdapter.getKeypointsFromNextCursorPosition();

        // Closes database
        MatchingDatabaseAdapter.closeDb(); 

        Features2d.drawKeypoints(rgba, keyPointsFromDb, rgba);
        Utils.matToBitmap(rgba, inputImage);
        imageView.setImageBitmap(inputImage);

    }
}

Here is the database code which contains some details associated with converting to the byte array required for the database. I didn't include everything associated with using a database, since that's really a different topic.

public class MatchingDatabaseAdapter {
    ...
    ...
    ...

    public static void addKeypoints(MatOfKeyPoint keyPoints, Context context) {
        float[] data = new float[(int)keyPoints.total() * keyPoints.channels()]; // make a spot to save the data
        keyPoints.get(0,0,data); // load the data;
        ByteBuffer buffer = ByteBuffer.allocate(data.length * 4);
        for (int i = 0; i < data.length; i++){
            buffer.putFloat(data[i]);
        }
        byte[] byteArray = buffer.array();
        addBlob(byteArray, keyPoints.rows(), keyPoints.cols(), keyPoints.type(), context);
    }

    public static void addBlob(byte[] blob, int rows, int columns, int mattype, Context context) throws SQLException {
        if (mDb == null) mDb = openDb(context);
        ContentValues contentValues = new ContentValues();
        contentValues.put(DatabaseHelper.BLOB_FIELD_NAME, blob);
        contentValues.put(DatabaseHelper.ROWS_FIELD_NAME, rows);
        contentValues.put(DatabaseHelper.COLUMNS_FIELD_NAME, columns);
        contentValues.put(DatabaseHelper.MATTYPE_FIELD_NAME, mattype);
        long x = mDb.insert(DatabaseHelper.TABLE_NAME, null, contentValues);
        Log.v(TAG, "insert said " + x + " and blob was " + blob.length + " long.");
        closeDb();
    }

    public static Cursor getAllRecordsCursor(Context context) {
        if (mDb == null || !mDb.isOpen()) mDb = openDb(context);
        mCursor = mDb.query(DatabaseHelper.TABLE_NAME, null, null, null, null,null, null);
        boolean hasRecords = mCursor.moveToFirst();
        Log.v(TAG, "MatchingDatabaseAdapter.getAllRecordsCursor() cursor created. " + mCursor + " and " + (hasRecords?"has records":"has NO RECORDS"));
        return mCursor;
    }

     public static MatOfKeyPoint getKeypointsFromNextCursorPosition() {
        MatOfKeyPoint keyPoints = null;
        if (mCursor != null) {
            Log.v(TAG, "mCursor has " + mCursor.getCount());
            mCursor.moveToFirst();
            int rows = mCursor.getInt(DatabaseHelper.ROWS_FIELD_POSITION);
            int columns = mCursor.getInt(DatabaseHelper.COLUMNS_FIELD_POSITION);
            int mattype = mCursor.getInt(DatabaseHelper.MATTYPE_FIELD_POSITION);
            keyPoints = new MatOfKeyPoint();
            keyPoints.create(rows, columns, mattype);
            byte[] blob = mCursor.getBlob(DatabaseHelper.BLOB_FIELD_POSITION);
            ByteBuffer buffer = ByteBuffer.wrap(blob);
            FloatBuffer floatBuffer = buffer.asFloatBuffer();
            float[] floatArray = new float[floatBuffer.limit()];
            floatBuffer.get(floatArray);
            keyPoints.put(0, 0, floatArray);
        }
        return keyPoints;
    }

    // INNER CLASS DatabaseHelper
    static class DatabaseHelper extends SQLiteOpenHelper {
        private static DatabaseHelper mDatabaseHelper;
        private static final String DB_NAME = "blobDb";
        private static final String TABLE_NAME = "blobTable";
        private static final String ROWS_FIELD_NAME = "rowsField";
        public static final int ROWS_FIELD_POSITION = 1;
        private static final String COLUMNS_FIELD_NAME = "columnsField";
        public static final int COLUMNS_FIELD_POSITION = 2;
        private static final String MATTYPE_FIELD_NAME = "mattypeField";
        public static final int MATTYPE_FIELD_POSITION = 3;
        private static final String BLOB_FIELD_NAME = "blobField";
        private static final int BLOB_FIELD_POSITION = 4;

        private static final java.lang.String CREATE_TABLE_SQL =
                "CREATE TABLE " + TABLE_NAME + " ("
                        + "_id INTEGER PRIMARY KEY AUTOINCREMENT, "
                        + ROWS_FIELD_NAME + " INTEGER, "
                        + COLUMNS_FIELD_NAME + " INTEGER, "
                        + MATTYPE_FIELD_NAME + " INTEGER, "
                        + BLOB_FIELD_NAME + " BLOB"
                        + ");";

        private DatabaseHelper(Context context) {
            super(context, DB_NAME, null, DB_VERSION);
            mDatabaseHelper = this;
        }

        static synchronized DatabaseHelper getInstance(Context context) {
            if (mDatabaseHelper == null) {
                mDatabaseHelper = new DatabaseHelper(context.getApplicationContext());
            }
            return mDatabaseHelper;
        }

        @Override
        public void onCreate(SQLiteDatabase db) {
            Log.v(TAG, "Creating table: " + CREATE_TABLE_SQL);
            db.execSQL(CREATE_TABLE_SQL);
        }

        @Override
        public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
            Log.v(TAG, "onUpgrade() from " + oldVersion + " to " + newVersion);
            Log.v(TAG, "ALL DATA BEING REMOVED FROM THE DATABASE!!");
            db.execSQL("DROP TABLE IF EXISTS " + TABLE_NAME + ";");
            onCreate(db);
        }
    }
}

The full sample project runs and displays the surf key points, so if you want to save keypoints to a database, this code should get you where you want to go.

Answer 3

Recently I complete a project about recognition human gesture. But I used svm from openCv. for your inform keypoint number in each image is its salient point (according to openCv definition). for recognition or in other words "prediction" you must save feature of keypoint (keypoint Property) that is a mat. I used SURF and it compute 64 feature for each keypoint. I hope it helps.