How to rotate a 3D tringle flat against the wall?

Question

I'm working with 3D mesh data, where I have lots of 3D triangles which I need to rotate to eliminate the Z value, converting it to a 2D triangle.

With this 2D triangle I'm doing some vector calculations.

After I'm done with my work I need to rotate it back to the original angle such that the old points return back to their original positions, to fit back into the 3D mesh.

---

Edit: This is the code I'm using.
I can't figure out how to reverse the rotation.

Inputs

var p1:Object, p2:Object, p3:Object;

Find face normal

var norm:Object = calcNormal(p1,p2,p3);

Find rotation angles based on normal

sinteta = -norm.y / Math.sqrt(norm.x * norm.x + norm.y * norm.y);
costeta = norm.x / Math.sqrt(norm.x * norm.x + norm.y * norm.y);
sinfi = -Math.sqrt(norm.x * norm.x + norm.y * norm.y);
cosfi = norm.z;

Rotate around Z and then Y to align to the z plane.

lx = costeta * cosfi;
ly = -sinteta * cosfi;
lz = sinfi;

mx = sinteta;
my = costeta;
mz = 0;

nx = -sinfi * costeta;
ny = sinfi * sinteta;
nz = cosfi;

var np1:Object = {};
np1.x=p1.x*lx + p1.y*ly + p1.z*lz;
np1.y=p1.x*mx + p1.y*my + p1.z*mz;
np1.z=p1.x*nx + p1.y*ny + p1.z*nz;

var np2:Object = {};
np2.x=p2.x*lx + p2.y*ly + p2.z*lz;
np2.y=p2.x*mx + p2.y*my + p2.z*mz;
np2.z=p2.x*nx + p2.y*ny + p2.z*nz;

var np3:Object = {};
np3.x=p3.x*lx + p3.y*ly + p3.z*lz;
np3.y=p3.x*mx + p3.y*my + p3.z*mz;
np3.z=p3.x*nx + p3.y*ny + p3.z*nz;

Answer 1

Determine the normal of the plane using the plane equation. Then, determine a quaternion that represents the rotation of the normal to the z axis. Rotate the polygon, do your work, and rotate it back.

A vector can be rotated by a quaternion by creating a quaternion from the vector where 'w' = 0:

v = (x, y, z) q = (w=0, x, y, z)

To rotate by q2,

rv = q2 * q * q2 ^ -1

To convert rv to a point, drop the w (which is 0).

To rotate back again, use

q2 ^ -1 * rv * q

where q2 ^ -1 is the inverse or conjugate of q2.

EDIT 2

Appologies for the C++ code, but here is how my Vector3d and Quaternion classes work (simplified):

class Vector3d {
  //...
  double x, y, z;
  //...
  // functions here e.g. dot (dot product), cross (cross product)
};

class Quaternion {
  //...
  double w, x, y, z;
  //...
  Quaternion inverse() const { // also equal to conjugate for unit quaternions
    return Quaternion (w, -x, -y, -z);
  }

  static Quaternion align(const Vector3d v1, const Vector3d v2) {
    Vector3d bisector = (v1 + v2).normalize();
    double cosHalfAngle = v1.dot(bisector);
    Vector3d cross;

    if(cosHalfAngle == 0.0) {
      cross = v1.cross(bisector);
    } else {
      cross = v1.cross(Vector3d(v2.z, v2.x, v2.y)).normalize();
    }

    return Quaternion(cosHalfAngle, cross.x, cross.y, cross.z);
  }

  Quaternion operator *(const Quaternion &q) const {
    Quaternion r;

    r.w = w * q.w - x * q.x - y * q.y - z * q.z;
    r.x = w * q.x + x * q.w + y * q.z - z * q.y;
    r.y = w * q.y + y * q.w + z * q.x - x * q.z;
    r.z = w * q.z + z * q.w + x * q.y - y * q.x;

    return r;
  }
};

So using this kind of maths, the idea is that you create a quaterion using the 'align' method which represents a rotation from the plane normal to the z axis (i.e. v1 is plane normal [normalized], v2 is z axis unit vector) - lets call that Q. To rotate each point, p, you would create a quaternion, q, for the point, rotate it, qr, then convert q back to a point, p2, like so:

q = Quaternion(0, p.x, p.y, p.z);
qr = Q * q * Q.inverse();
p2 = Vector3d(qr.x, qr.y, qr.z);

To rotate p2 back again, do:

q = Quaternion(0, p2.x, p2.y, p2.z);
qr = Q.inverse() * q * Q;
p = Vector3d(qr.x, qr.y, qr.z);

Answer 2

Rotating the triangle is almost certainly the wrong approach. If your intent is to modify the triangle, then you should just perform the operations in its current (3d) coordinate system. If your intent is not to modify the triangle then you don't need to rotate it back. If you don't know how to perform the operations you want to in its current coordinate system, then ask that question.