How to match 3D perspective of real photo and obje

Question

The situation:

In my particular case I have 3D photoframe image and a 2D photo. And I want the 2D photo to match the 3D frame using CSS3 transform functions (Rotate, Scale, Skew).

The problem:

I wasn't able to precisely match the two using manual method aka typing rotation value and watching what it does.

Ideal solution #1

Online visual tool exists that lets you drag corners of the photo (like photoshop does) and It gives you the correct CSS3 transform values.

Ideal solution #2

Non-visual tool exist - same as before but you manually enter 4 point coordinates (image corners) and it gives you the correct CSS3 transform values.

Real solution of this question

If there aren't such tools (my search found none) I would like somebody to try explain the math behind it so I could calculate it myself - If it is even possible?

I prepared JSFiddle demo for you fiddle around: Demo

/* Main issue here */

.transform {
  transform: rotateX(34deg) rotateZ(13deg) rotateY(-10deg) scaleY(1) scaleX(1) skewY(0deg) skewX(0deg) translateY(0px) translateX(20px);
  transform-origin: 50% 0% 0;
}
/* Supporting styles */

.container {
  position: relative;
  width: 500px;
  height: 500px;
}
.frame,
.photo {
  position: absolute;
  top: 0;
  left: 0;
  right: 0;
  bottom: 0;
}
.photo {
  top: 50px;
  left: 95px;
  right: 65px;
  bottom: 270px;
}
.frame img,
.photo img {
  width: 100%
}
.frame {
  z-index: 2;
}

<div class="container">

  <div class="frame">
    <img src="http://cdn.idesigned.cz/img/cc08acc7b9b08ab53bf935d720210f13.png" />
  </div>

  <div class="photo">
    <div class="transform">
      <img src="https://static.pexels.com/photos/7976/pexels-photo.jpg" />
    </div>
  </div>

</div>

Answer 1

I can imagine that it is difficult to find a tool or formula for this. If you know the angles and measures of the book it is easy to calculate I think. W3C Working Draft here you can find more detailed information about the transformations but as I said, without the details of your image (the book) I have no idea how you could calculate the exact coordinates.

The only thing that could help you is the css perspective. I have made a Plunk that you can see how it looks like with it.

The only things that I changed are:

.container {
  position: relative;
  width: 500px;
  height: 500px;
  perspective: 500px;
}

and

.transform{
    -webkit-transform: rotateX(19deg) rotateZ(6deg) rotateY(0deg) scaleY(0.85) scaleX(0.85) skewY(0deg) skewX(-8deg) translateY(-10px) translateX(33px);
    -webkit-transform-origin: 50% 0% 0;
    transform: rotateX(19deg) rotateZ(6deg) rotateY(0deg) scaleY(0.85) scaleX(0.85) skewY(0deg) skewX(-8deg) translateY(-10px) translateX(33px);
    transform-origin: 50% 0% 0;
 }

The photo doesn't fit exactly in the frame because it's a square but I hope that helps you a bit further.

Answer 2

If you can use 3-d transforms (such as rotateZ), then you can also provide matrix3d which you can compute from desired point correspondences.

Here's a fiddle: https://jsfiddle.net/szym/03s5mwjv/

I'm using numeric.js to solve a set of 4 linear equations to find the perspective transform matrix that transforms src onto dst. This is essentially the same math as in getPerspectiveTransform in OpenCV.

The computed 2-d perspective transform is a 3x3 matrix using homogeneous coordinates. The CSS matrix3d is a 4x4 matrix using homogeneous coordinates, so we need to add an identity row/column for the z axis. Furthermore, matrix3d is specified in column-major order.

Once you get the matrix3d you can just paste it into your stylesheet. But keep in mind that the matrix is computed assuming (0, 0) as origin, so you also need to set transformOrigin: 0 0.

// Computes the matrix3d that maps src points to dst.
function computeTransform(src, dst) {
  // src and dst should have length 4 each
  var count = 4;
  var a = []; // (2*count) x 8 matrix
  var b = []; // (2*count) vector

  for (var i = 0; i < 2 * count; ++i) {
    a.push([0, 0, 0, 0, 0, 0, 0, 0]);
    b.push(0);
  }

  for (var i = 0; i < count; ++i) {
    var j = i + count;
    a[i][0] = a[j][3] = src[i][0];
    a[i][1] = a[j][4] = src[i][1];
    a[i][2] = a[j][5] = 1;
    a[i][3] = a[i][4] = a[i][5] =
      a[j][0] = a[j][1] = a[j][2] = 0;
    a[i][6] = -src[i][0] * dst[i][0];
    a[i][7] = -src[i][1] * dst[i][0];
    a[j][6] = -src[i][0] * dst[i][1];
    a[j][7] = -src[i][1] * dst[i][1];
    b[i] = dst[i][0];
    b[j] = dst[i][1];
  }

  var x = numeric.solve(a, b);
  // matrix3d is homogeneous coords in column major!
  // the z coordinate is unused
  var m = [
    x[0], x[3],   0, x[6],
    x[1], x[4],   0, x[7],
       0,    0,   1,    0,
    x[2], x[5],   0,    1
  ];
  var transform = "matrix3d(";
  for (var i = 0; i < m.length - 1; ++i) {
    transform += m[i] + ", ";
  }
  transform += m[15] + ")";
  return transform;
}

// Collect the four corners by user clicking in the corners
var dst = [];
document.getElementById('frame').addEventListener('mousedown', function(evt) {
  // Make sure the coordinates are within the target element.
  var box = evt.target.getBoundingClientRect();
  var point = [evt.clientX - box.left, evt.clientY - box.top];
  dst.push(point);

  if (dst.length == 4) {
    // Once we have all corners, compute the transform.
    var img = document.getElementById('img');
    var w = img.width,
        h = img.height;
    var transform = computeTransform(
      [
        [0, 0],
        [w, 0],
        [w, h],
        [0, h]
      ],
      dst
    );
    document.getElementById('photo').style.visibility = 'visible';
    document.getElementById('transform').style.transformOrigin = '0 0';
    document.getElementById('transform').style.transform = transform;
    document.getElementById('result').innerHTML = transform;
  }
});

.container {
  position: relative;
  width: 50%;
}
  
#frame,
#photo {
  position: absolute;
  top: 0;
  left: 0;
  right: 0;
  bottom: 0;
}
  
#photo {
  visibility: hidden;
}
  
#frame img,
#photo img {
  width: 100%
}
  
#photo {
  opacity: 0.7;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/numeric/1.2.6/numeric.min.js"></script>
  <p id="result">Click the desired top-left, top-right, bottom-right, bottom-left corners
  <div class="container">
    <div id="frame">
      <img src="http://cdn.idesigned.cz/img/cc08acc7b9b08ab53bf935d720210f13.png" />
    </div>

    <div id="photo">
      <div id="transform">
        <img id="img" src="http://placehold.it/350x150" />
      </div>
    </div>

  </div>

Answer 3

I've written an answer on Math SE about how to compute the transformation matrix to map the corners of one image to four given coordinates using a projective transformation. It has details of what's actually going on in that computation. It also has some CSS and I adapted the original demo to your example scenario:

function adj(m) { // Compute the adjugate of m
  return [
    m[4]*m[8]-m[5]*m[7], m[2]*m[7]-m[1]*m[8], m[1]*m[5]-m[2]*m[4],
    m[5]*m[6]-m[3]*m[8], m[0]*m[8]-m[2]*m[6], m[2]*m[3]-m[0]*m[5],
    m[3]*m[7]-m[4]*m[6], m[1]*m[6]-m[0]*m[7], m[0]*m[4]-m[1]*m[3]
  ];
}
function multmm(a, b) { // multiply two matrices
  var c = Array(9);
  for (var i = 0; i != 3; ++i) {
    for (var j = 0; j != 3; ++j) {
      var cij = 0;
      for (var k = 0; k != 3; ++k) {
        cij += a[3*i + k]*b[3*k + j];
      }
      c[3*i + j] = cij;
    }
  }
  return c;
}
function multmv(m, v) { // multiply matrix and vector
  return [
    m[0]*v[0] + m[1]*v[1] + m[2]*v[2],
    m[3]*v[0] + m[4]*v[1] + m[5]*v[2],
    m[6]*v[0] + m[7]*v[1] + m[8]*v[2]
  ];
}
function basisToPoints(x1, y1, x2, y2, x3, y3, x4, y4) { // map basis to these points
  var m = [
    x1, x2, x3,
    y1, y2, y3,
     1,  1,  1
  ];
  var v = multmv(adj(m), [x4, y4, 1]);
  return multmm(m, [
    v[0], 0, 0,
    0, v[1], 0,
    0, 0, v[2]
  ]);
}
function general2DProjection(
  x1s, y1s, x1d, y1d,
  x2s, y2s, x2d, y2d,
  x3s, y3s, x3d, y3d,
  x4s, y4s, x4d, y4d
) {
  console.log(Array.prototype.join.call(arguments, ", "));
  var s = basisToPoints(x1s, y1s, x2s, y2s, x3s, y3s, x4s, y4s);
  var d = basisToPoints(x1d, y1d, x2d, y2d, x3d, y3d, x4d, y4d);
  return multmm(d, adj(s));
}
function transform2d(elt, x1, y1, x2, y2, x3, y3, x4, y4) {
  var w = elt.offsetWidth, h = elt.offsetHeight;
  var t = general2DProjection
    (0, 0, x1, y1, w, 0, x2, y2, 0, h, x3, y3, w, h, x4, y4);
  for(i = 0; i != 9; ++i) t[i] = t[i]/t[8];
  t = [t[0], t[3], 0, t[6],
       t[1], t[4], 0, t[7],
       0   , 0   , 1, 0   ,
       t[2], t[5], 0, t[8]];
  t = "matrix3d(" + t.join(", ") + ")";
  elt.style["-webkit-transform"] = t;
  elt.style["-moz-transform"] = t;
  elt.style["-o-transform"] = t;
  elt.style.transform = t;
}

corners = [100, 50, 300, 50, 100, 150, 300, 150];
function update() {
  var box = document.getElementById("photo");
  transform2d(box, corners[0], corners[1], corners[2], corners[3],
                   corners[4], corners[5], corners[6], corners[7]);
  for (var i = 0; i != 8; i += 2) {
    var elt = document.getElementById("marker" + i);
    elt.style.left = corners[i] + "px";
    elt.style.top = corners[i + 1] + "px";
  }
  document.getElementById("matrix").textContent = box.style.transform;
}
function move(evnt) {
  if (currentcorner < 0) return;
  corners[currentcorner] = evnt.pageX;
  corners[currentcorner + 1] = evnt.pageY;
  console.log(corners);
  update();
}
currentcorner = -1;
window.addEventListener('load', function() {
  document.documentElement.style.margin="0px";
  document.documentElement.style.padding="0px";
  document.body.style.margin="0px";
  document.body.style.padding="0px";
  update();
});
window.addEventListener('mousedown', function(evnt) {
  var x = evnt.pageX, y = evnt.pageY, dx, dy;
  var best = 400; // 20px grab radius
  currentcorner = -1;
  for (var i = 0; i != 8; i += 2) {
    dx = x - corners[i];
    dy = y - corners[i + 1];
    if (best > dx*dx + dy*dy) {
      best = dx*dx + dy*dy;
      currentcorner = i;
    }
  }
  move(evnt);
  evnt.preventDefault();
}, true);
window.addEventListener('mouseup', function(evnt) {
  currentcorner = -1;
}, true)
window.addEventListener('mousemove', move, true);

/* Supporting styles */

#photo {
  position: absolute;
  z-index: 1;
  transform-origin: 0% 0% 0;
}
.dot {
  position: absolute;
  z-index: 2;
  margin: -0.5ex;
  padding: 0ex;
  width: 1ex;
  height: 1ex;
  border-radius: 0.5ex;
  background-color: #ff0000;
}

<img id="photo" src="https://static.pexels.com/photos/7976/pexels-photo.jpg" />
<img class="frame" src="http://cdn.idesigned.cz/img/cc08acc7b9b08ab53bf935d720210f13.png" />
<div class="dot" id="marker0"></div>
<div class="dot" id="marker2"></div>
<div class="dot" id="marker4"></div>
<div class="dot" id="marker6"></div>
<div id="matrix"></div>

The formulation is such that you can make it working with merely three arithmetic operations: +, - and *. You don't even need / (if you use adjunct instead of inverse matrices), much less case distinctions, square roots, or any other such things.

If you prefer Stack Overflow (and to get a cross reference established), see Redraw image from 3d perspective to 2d.