Work with an Amazon S3 response stream after respo

Question

I'm using the Amazon SDK and I have a method that returns a Stream for an object stored in Amazon S3 service.

It contains something like this:

var request = new GetObjectRequest().WithBucketName(bucketName).WithKey(keyName);
using (var response = client.GetObject(request))
{
    return response.ResponseStream;
}

Obviously, when doing this, the stream is not readable from the calling method because the request object has been disposed and when this is done, it closes the stream.

I don't want to download the file to a MemoryStream or a FileStream.

If I don't use the using clause, the garbage collector will dispose the request object at some point so I can't just not use it.

What I'm asking is, is there a way to wrap or copy the Stream into another Stream and then return it without having to download the file?

I'm using .NET 3.5.

Edit: The method is inherited from an abstract class and the caller method doesn't know it is working with Amazon. So it HAS to return a Stream.

Answer 1

For completeness, and since I went with @Samuel option number 2 (Wrap the stream) which @spakinz commented he did too, I include here my implementation of what I called AmazonS3Stream

public class AmazonS3Stream : Stream
{
    private Stream stream;
    private GetObjectResponse response;

    public AmazonS3Stream(GetObjectResponse response)
    {
        this.stream = response.ResponseStream;
        this.response = response;
    }

    // The whole purpose of this class
    protected override void Dispose(bool disposing)
    {
        // base.Dispose(disposing); // Do we really need this line? Probably not since I tested it and I can see that the stream is disposed when Response.Dispose is called by itself. And that makes sense because we know that this.stream is pointing to response.ResponseStream (that's what we declared in the constructor: this.stream = response.ResponseStream; ) So, what do we expect from response.Dispose() ? Obviously the first thing it would do is call ResponseStream.Dispose()
        response.Dispose();
    }

    public override long Position
    {
        get { return stream.Position; }
        set { stream.Position = Position; }
    }

    public override long Length
    {
        get { return stream.Length; }
    }

    public override bool CanRead
    {
        get { return stream.CanRead; }
    }

    public override bool CanSeek
    {
        get { return stream.CanSeek; }
    }

    public override bool CanWrite
    {
        get { return stream.CanWrite; }
    }

    public override void Flush()
    {
        stream.Flush();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        stream.Write(buffer, offset, count);
    }

    public override void SetLength(long value)
    {
        stream.SetLength(value);
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        return stream.Seek(offset, origin);
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        return stream.Read(buffer, offset, count);
    }
}

Answer 2

If your function returns the response object (without using the using statement), and the caller assigns it to a variable, there will still be a reference to the response object. Thus it won't be eligible garbage collected.

Answer 3

Had the same issue (I thought). But have you tried to not use "using". This will not use the stream and send it to the caller who will have the responsibility to dispose it. As simple as that.

    var request = new GetObjectRequest { BucketName = containerName, Key = blobName };
    GetObjectResponse response = null;

        try
        {
            response = client.GetObject(request));
        }
        catch (AmazonS3Exception ex)
        {
            if ((ex.ErrorCode == "NoSuchBucket") || (ex.ErrorCode == "AccessDenied") || (ex.ErrorCode == "InvalidBucketName") || (ex.ErrorCode == "NoSuchKey"))
            {
                return null;
            }

            throw;
        }

        return response.ResponseStream;

Answer 4

You can't work with the stream after it's disposed, but you can postpone disposing the response object until after the response stream has been used. There are three options I can suggest.

1. Return Response. One option is to return the response object to the caller. The caller can access the contained response stream and then dispose the response when done. This is the easiest change but requires the caller to change as well.

2. Wrap the stream. Instead of returning the response stream directly, create a new object that extends Stream and wraps the response stream but also has a reference to the response itself. Then when your wrapper is disposed, you can internally dispose the response object. This requires only changing your called code as long as the type being returned is just Stream.

3. Callback. Don't return anything from the method and use a callback mechanism instead. Accept an Action<Stream> as a parameter and call this callback with the stream. That way the callers code is called and you still have control over disposing the response and stream. This is the safest mechanism since you're not relying on the caller to dispose anything, but requires the most changes.

Answer 5

To add a code example to @Jun Y.'s answer. This is what I did and it was the cleanest solution by far.

using (var client = new AmazonS3Client(Amazon.RegionEndpoint.USEast2))
{
    var transferUtility = new TransferUtility(client);
    return await transferUtility.OpenStreamAsync(S3BucketName, key);
}

Answer 6

There is a method OpenStream in TransferUtility class which returns a stream object.

public Stream OpenStream( String bucketName, String key )

I looked through the source code of AWSSDK, and found in the SDK the OpenStream just returns the response stream directly. (It doesn't use "using" statement on response object.)