Is there a way to display JPEG image via MTKView and MTLBuffer(and within iPhone 6+). I've tried this in a following way(this was just test):
- (id<MTLBuffer>)testBuffer
{
if (!_testBuffer)
{
//
NSString *path = [[NSBundle mainBundle] pathForResource:@"testImage" ofType:@"jpg"];
NSData *imageData = [NSData dataWithContentsOfFile:path];
_testBuffer = [self.device newBufferWithBytes:imageData.bytes length:imageData.length options:MTLResourceCPUCacheModeWriteCombined];
}
return _testBuffer;
}
- (void)drawInMTKView:(MTKView *)view
{
MTLRenderPassDescriptor* renderPassDescriptor = view.currentRenderPassDescriptor;
id <MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer];
id <MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
[renderEncoder drawPrimitives:MTLPrimitiveTypePoint indirectBuffer:self.testBuffer indirectBufferOffset:0];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:view.currentDrawable];
[commandBuffer commit];
}
I get an error in this method call:
[renderEncoder drawPrimitives:MTLPrimitiveTypePoint indirectBuffer:self.testBuffer indirectBufferOffset:0];
/BuildRoot/Library/Caches/com.apple.xbs/Sources/Metal/Metal-54.31/ToolsLayers/Debug/MTLDebugRenderCommandEncoder.mm:2123:
failed assertion `-[MTLDebugRenderCommandEncoder
drawPrimitives:indirectBuffer:indirectBufferOffset:] is only supported
on MTLFeatureSet_iOS_GPUFamily3_v1 and later'
This is because this API supported only by iPhone 6s(+).
And I think I am doing something completely wrong. Maybe I need to think in other direction. Could somebody help me or point to right direction?
Thanks.
There is no simple way to use MTLBuffer to present image data via MTKView. In this case you have to prepare right pixel presentation for right output texture that is used to display by MTKView in the drawble layer. The simplest way is to use MTKTextureLoader, or load data to MTLTexture object from UIImage/CGImage instance.
I did a similar experiments with presenting images on iOS devices using MTL Layer an MTLTexture (instead of MTLBuffer), and i've decided the best way to display image (if this one needs only: to display image) it uses passthrough kernel-function without rendering.
In Swift2 this looks like the follow:
...
let textureLoader = MTKTextureLoader(device: self.device!)
if let image = UIImage(named: file){
imageTexture = try! textureLoader.newTextureWithCGImage(image.CGImage!, options: nil)
threadGroups = MTLSizeMake(
(imageTexture.width+threadGroupCount.width)/threadGroupCount.width,
(imageTexture.height+threadGroupCount.height)/threadGroupCount.height, 1)
}
}
...
let function:MTLFunction! = library.newFunctionWithName("kernel_passthrough")
...
pipeline = try! self.device.newComputePipelineStateWithFunction(function)
...
let commandBuffer = commandQueue.commandBuffer()
let encoder = commandBuffer.computeCommandEncoder()
encoder.setComputePipelineState(pipeline)
encoder.setTexture(actualImageTexture, atIndex: 0)
encoder.setTexture(metalView.currentDrawable!.texture, atIndex: 1)
encoder.setBuffer(self.saturationUniform, offset: 0, atIndex: 0)
encoder.dispatchThreadgroups(threadGroups!, threadsPerThreadgroup: threadGroupCount)
encoder.endEncoding()
commandBuffer.presentDrawable(metalView.currentDrawable!)
commandBuffer.commit()
...
In Metal Shading File:
kernel void kernel_passthrough(texture2d<float, access::read> inTexture [[texture(0)]],
texture2d<float, access::write> outTexture [[texture(1)]],
uint2 gid [[thread_position_in_grid]])
{
float4 inColor = inTexture.read(gid);
//
// flip texture vertically if it needs to display with right orientation
//
outTexture.write(inColor, gid);
}
Full example sources: ImageMetalling-00
I’ve forgotten to write solution due to being busy. The @Denn's answer looks like correct so I’ll accept it.
My solution below(in objective-c). I’ve simplified apple examples(as well as their shader file) and made test project.
I’ve created subclass of MTKView called MetalView. It has ‘configure' method, which prepares MetalView for drawing: sets MTKView’s properties and creates pipelineState object as well as vertex buffers.
- (void)configure
{
self.device = MTLCreateSystemDefaultDevice();
self.colorPixelFormat = MTLPixelFormatBGRA8Unorm;
self.framebufferOnly = YES;
self.sampleCount = 1;
// create command queue for usage during drawing
self.commandQueue = [self.device newCommandQueue];
// load shaders functions from texturedQuad.metal file. These functions needed for configuration of MTLRenderPipelineDescriptor
id <MTLLibrary> shaderLibrary = [self.device newDefaultLibrary];
id <MTLFunction> fragmentProgram = [shaderLibrary newFunctionWithName:@"texturedQuadFragment"];
id <MTLFunction> vertexProgram = [shaderLibrary newFunctionWithName:@"texturedQuadVertex"];
// create a pipeline state
MTLRenderPipelineDescriptor *pQuadPipelineStateDescriptor = [MTLRenderPipelineDescriptor new];
pQuadPipelineStateDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
pQuadPipelineStateDescriptor.sampleCount = self.sampleCount;
pQuadPipelineStateDescriptor.vertexFunction = vertexProgram;
pQuadPipelineStateDescriptor.fragmentFunction = fragmentProgram;
NSError *pipErr = nil;
self.pipelineState = [self.device newRenderPipelineStateWithDescriptor:pQuadPipelineStateDescriptor
error:&pipErr];
if (pipErr) NSLog(@"newRenderPipelineStateWithDescriptor err: %@", pipErr);
// buffers for MTLRenderCommandEncoder in drawRect: method
self.vertexBuffer = [self.device newBufferWithBytes:kQuadVertices
length:kSzQuadVertices
options:MTLResourceOptionCPUCacheModeDefault];
self.texCoordBuffer = [self.device newBufferWithBytes:kQuadTexCoords
length:kSzQuadTexCoords
options:MTLResourceOptionCPUCacheModeDefault];
self.paused = YES;
self.enableSetNeedsDisplay = NO;
}
These stuff will be used by renderEncoder in drawRect: method for actual drawing:
- (void)drawRect:(CGRect)rect
{
id <MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
MTLRenderPassDescriptor *renderPassDescriptor = self.currentRenderPassDescriptor;
id <MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
// Encode into a renderer
[renderEncoder setRenderPipelineState:self.pipelineState];
[renderEncoder setVertexBuffer:self.vertexBuffer
offset:0
atIndex:0];
[renderEncoder setVertexBuffer:self.texCoordBuffer
offset:0
atIndex:1];
[renderEncoder setFragmentTexture:self.texture
atIndex:0];
// tell the render context we want to draw our primitives. We will draw triangles that's
// why we need kQuadVertices and kQuadTexCoords (arrays of points)
[renderEncoder drawPrimitives:MTLPrimitiveTypeTriangle
vertexStart:0
vertexCount:6
instanceCount:1];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:self.currentDrawable];
[commandBuffer commit];
}
Shader file code(texturedQuad.metal):
struct VertexInOut
{
float4 m_Position [[position]];
float2 m_TexCoord [[user(texturecoord)]];
};
vertex VertexInOut texturedQuadVertex(constant float4 *pPosition [[ buffer(0) ]],
constant packed_float2 *pTexCoords [[ buffer(1) ]],
constant float4x4 *pMVP [[ buffer(2) ]],
uint vid [[ vertex_id ]])
{
VertexInOut outVertices;
outVertices.m_Position = pPosition[vid];
outVertices.m_TexCoord = pTexCoords[vid];
return outVertices;
}
fragment half4 texturedQuadFragment(VertexInOut inFrag [[ stage_in ]],
texture2d<half> tex2D [[ texture(0) ]])
{
constexpr sampler quad_sampler;
half4 color = tex2D.sample(quad_sampler, inFrag.m_TexCoord);
return color;
}
MetalView class is used as class for view outlet in ViewController’s storyboard. ViewController’s viewDidLoad method retrieves object for specified image and sets this object to MetalView.
- (void)viewDidLoad
{
[super viewDidLoad];
self.metalView = (MetalView *)self.view;
NSString *path = [[NSBundle mainBundle] pathForResource:@"testImage"
ofType:@"jpg"];
MTKTextureLoader *loader = [[MTKTextureLoader alloc] initWithDevice:self.metalView.device];
NSError *err = nil;
self.metalView.texture = [loader newTextureWithContentsOfURL:[NSURL fileURLWithPath:path] options:nil error:&err];
if (err) NSLog(@"newTextureWithContentsOfURL err: %@", [err localizedDescription]);
}
And finally, we need to call draw method for MetalView
- (void)viewDidLayoutSubviews
{
[self.metalView draw];
}
Sample code is here.
