How to generate an UIImage from AVCapturePhoto wit

Question

I am calling AVFoundation's delegate method to handle a photo capture, but I am having difficulty converting the AVCapturePhoto it generates into an UIImage with the correct orientation. Although the routine below is successful, I always get a right-oriented UIImage (UIImage.imageOrientation = 3). I have no way of providing an orientation when using the UIImage(data: image) and attempting to first use photo.cgImageRepresentation()?.takeRetainedValue() also doesn't help. Please assist.

Image orientation is critical here as the resulting image is being fed to a Vision Framework workflow.

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    // capture image finished
    print("Image captured.")
    if let imageData = photo.fileDataRepresentation() {
        if let uiImage = UIImage(data: imageData){
            // do stuff to UIImage
        }
    }
}

UPDATE 1: Reading Apple's Photo Capture Programming Guide (out of date for iOS11), I did manage to find one thing I was doing wrong:

1. On every capture call (self.capturePhotoOutput.capturePhoto) one must setup a connection with the PhotoOutput object and update its orientation to match the device's orientation at the moment the picture is taken. For doing that, I created an extension of UIDeviceOrientation and used it on the snapPhoto() function I created to call the capture routine and wait for the didFinishProcessingPhoto delegate method to be executed. I've added a snapshot of the code because the code sample delimiters here don't seem to be displaying them correctly.

Update 2 Link to full project on GitHub: https://github.com/agu3rra/Out-Loud

Answer 1

Updated extension provided by Andre which works with Swift 4.2:

import Foundation
import UIKit

extension UIDeviceOrientation {
    var imageOrientation: UIImage.Orientation {
        switch self {
        case .portrait, .faceUp:                return .right
        case .portraitUpsideDown, .faceDown:    return .left
        case .landscapeLeft:                    return .up
        case .landscapeRight:                   return .down
        case .unknown:                          return .up
        }
    }
}

Answer 2

I've had success doing this:

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {

        let cgImage = photo.cgImageRepresentation()!.takeRetainedValue()
        let orientation = photo.metadata[kCGImagePropertyOrientation as String] as! NSNumber
        let uiOrientation = UIImage.Orientation(rawValue: orientation.intValue)!
        let image = UIImage(cgImage: cgImage, scale: 1, orientation: uiOrientation)

}

It's based on what Apple mention in their docs:

Each time you access this method, AVCapturePhoto generates a new CGImageRef. When backed by a compressed container (such as HEIC), the CGImageRepresentation is decoded lazily as needed. When backed by an uncompressed format such as BGRA, it is copied into a separate backing buffer whose lifetime is not tied to that of the AVCapturePhoto. For a 12 megapixel image, a BGRA CGImage represents ~48 megabytes per call. If you only intend to use the CGImage for on-screen rendering, use the previewCGImageRepresentation instead. Note that the physical rotation of the CGImageRef matches that of the main image. Exif orientation has not been applied. If you wish to apply rotation when working with UIImage, you can do so by querying the photo's metadata[kCGImagePropertyOrientation] value, and passing it as the orientation parameter to +[UIImage imageWithCGImage:scale:orientation:]. RAW images always return a CGImageRepresentation of nil. If you wish to make a CGImageRef from a RAW image, use CIRAWFilter in the CoreImage framework.

Answer 3

To create our image with the right orientation we need to enter the correct UIImage.Orientation when we initialize the image.

Its best to use the CGImagePropertyOrientation that comes back from the photoOutput delegate to get the exact orientation the camera session was in when the picture was taken. Only problem here is that while the enum values between UIImage.Orientation and CGImagePropertyOrientation are the same, the raw values are not. Apple suggests a simple mapping to fix this.

https://developer.apple.com/documentation/imageio/cgimagepropertyorientation

Here is my implementation:

AVCapturePhotoCaptureDelegate

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        if let _ = error {
            // Handle Error
        } else if let cgImageRepresentation = photo.cgImageRepresentation(),
            let orientationInt = photo.metadata[String(kCGImagePropertyOrientation)] as? UInt32,
            let imageOrientation = UIImage.Orientation.orientation(fromCGOrientationRaw: orientationInt) {

            // Create image with proper orientation
            let cgImage = cgImageRepresentation.takeUnretainedValue()
            let image = UIImage(cgImage: cgImage,
                                scale: 1,
                                orientation: imageOrientation)
        }
    }

Extension for Mapping

extension UIImage.Orientation {

    init(_ cgOrientation: CGImagePropertyOrientation) {
        // we need to map with enum values becuase raw values do not match
        switch cgOrientation {
        case .up: self = .up
        case .upMirrored: self = .upMirrored
        case .down: self = .down
        case .downMirrored: self = .downMirrored
        case .left: self = .left
        case .leftMirrored: self = .leftMirrored
        case .right: self = .right
        case .rightMirrored: self = .rightMirrored
        }
    }


    /// Returns a UIImage.Orientation based on the matching cgOrientation raw value
    static func orientation(fromCGOrientationRaw cgOrientationRaw: UInt32) -> UIImage.Orientation? {
        var orientation: UIImage.Orientation?
        if let cgOrientation = CGImagePropertyOrientation(rawValue: cgOrientationRaw) {
            orientation = UIImage.Orientation(cgOrientation)
        } else {
            orientation = nil // only hit if improper cgOrientation is passed
        }
        return orientation
    }
}

Answer 4

Final update: I ran some experiments with the app and came to the following conclusions:

1. kCGImagePropertyOrientation doesn’t seem to influence the orientation of the captured image inside your application and it only varies with the device orientation if you update your photoOutput connection each time you are about to call the capturePhoto method. So:

   func snapPhoto() {
       // prepare and initiate image capture routine
   
       // if I leave the next 4 lines commented, the intented orientation of the image on display will be 6 (right top) - kCGImagePropertyOrientation
       let deviceOrientation = UIDevice.current.orientation // retrieve current orientation from the device
       guard let photoOutputConnection = capturePhotoOutput.connection(with: AVMediaType.video) else {fatalError("Unable to establish input>output connection")}// setup a connection that manages input > output
       guard let videoOrientation = deviceOrientation.getAVCaptureVideoOrientationFromDevice() else {return}
       photoOutputConnection.videoOrientation = videoOrientation // update photo's output connection to match device's orientation
   
       let photoSettings = AVCapturePhotoSettings()
       photoSettings.isAutoStillImageStabilizationEnabled = true
       photoSettings.isHighResolutionPhotoEnabled = true
       photoSettings.flashMode = .auto
       self.capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self) // trigger image capture. It appears to work only if the capture session is running.
   }
   

2. Viewing the generated images on the debugger has shown me how they get generated, so I could infer the required rotation (UIImageOrientation) to get it displayed upright. In other words: updating UIImageOrientation tells how the image should be rotated in order for you to see it in the correct orientation. So I came to the following table:

3. I had to update my UIDeviceOrientation extension to a rather unintuitive form:

   extension UIDeviceOrientation {
       func getUIImageOrientationFromDevice() -> UIImageOrientation {
           // return CGImagePropertyOrientation based on Device Orientation
           // This extented function has been determined based on experimentation with how an UIImage gets displayed.
           switch self {
           case UIDeviceOrientation.portrait, .faceUp: return UIImageOrientation.right
           case UIDeviceOrientation.portraitUpsideDown, .faceDown: return UIImageOrientation.left
           case UIDeviceOrientation.landscapeLeft: return UIImageOrientation.up // this is the base orientation
           case UIDeviceOrientation.landscapeRight: return UIImageOrientation.down
           case UIDeviceOrientation.unknown: return UIImageOrientation.up
           }
       }
   }
   

4. This is how my final delegate method looks now. It displays the image in the expected orientation.

   func photoOutput(_ output: AVCapturePhotoOutput,
                                    didFinishProcessingPhoto photo: AVCapturePhoto,
                                    error: Error?)
   {
       // capture image finished
       print("Image captured.")
   
       let photoMetadata = photo.metadata
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       //                print("Metadata orientation: ",photoMetadata["Orientation"])
   
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       print("Metadata orientation with key: ",photoMetadata[String(kCGImagePropertyOrientation)] as Any)
   
       guard let imageData = photo.fileDataRepresentation() else {
           print("Error while generating image from photo capture data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       guard let uiImage = UIImage(data: imageData) else {
           print("Unable to generate UIImage from image data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       // generate a corresponding CGImage
       guard let cgImage = uiImage.cgImage else {
           print("Error generating CGImage");self.lastPhoto=nil;return
   
       }
   
       guard let deviceOrientationOnCapture = self.deviceOrientationOnCapture else {
           print("Error retrieving orientation on capture");self.lastPhoto=nil;
           return
   
       }
   
       self.lastPhoto = UIImage(cgImage: cgImage, scale: 1.0, orientation: deviceOrientationOnCapture.getUIImageOrientationFromDevice())
   
       print(self.lastPhoto)
       print("UIImage generated. Orientation:(self.lastPhoto.imageOrientation.rawValue)")
       self.controller.goToProcessing()
   }
   
   
   func photoOutput(_ output: AVCapturePhotoOutput, 
                      willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) 
                      {
       print("Just about to take a photo.")
       // get device orientation on capture
       self.deviceOrientationOnCapture = UIDevice.current.orientation
       print("Device orientation: \(self.deviceOrientationOnCapture.rawValue)")
   }
   

Answer 5

Inside the AVCapturePhoto I’m pretty sure you will find a metadata object of the also called CGImageProperties.
Inside it you will find the EXIF dictionary for orientation the next step is just to take the orientation and create an image according to that.
I do not have experiences in using AVCapturePhotoOutput but I have some using the old way.
Pay attention that the EXIF dictionary is mapped differently in UIImageOrientation.
Here is an article I wrote a lot of time ago, but the main principle are still valid.
This question will point you to some implementations, it's pretty old too, I'm pretty sure that in the latest version they released easier API, but it will still guide you into take the issue.