我想用Gzip压缩在Java中压缩的输入流。
比方说，我们有一个输入流（1GB的数据。）没有压缩。 我想作为一个结果，从源头上压缩的InputStream：
public InputStream getCompressedStream(InputStream unCompressedStream) {
// Not working because it's uncompressing the stream, I want the opposite.
return new GZIPInputStream(unCompressedStream);
}
DeflaterInputStream是不是你想要的，因为它缺乏gzip头/尾，并使用一个稍微不同的压缩。
如果您的OutputStream（推）以InputStream的（拉）改变你需要做的事情不同。
什么GzipOutputStream所做的是：
如果你想要做InputStreams一样，你需要一个包含流：
要做到这一点，最好的办法是提供3个不同的数据流，并将其合并为一个。 幸运的是的SequenceInputStream，做流的结合为您服务。
下面是我实现加一个简单的单元测试：
import java.io.ByteArrayInputStream;
import java.io.FileInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.SequenceInputStream;
import java.util.Enumeration;
import java.util.zip.CRC32;
import java.util.zip.Deflater;
import java.util.zip.DeflaterInputStream;
import java.util.zip.DeflaterOutputStream;
/**
* @author mwyraz
* Wraps an input stream and compresses it's contents. Similiar to DeflateInputStream but adds GZIP-header and trailer
* See GzipOutputStream for details.
* LICENSE: Free to use. Contains some lines from GzipOutputStream, so oracle's license might apply as well!
*/
public class GzipCompressingInputStream extends SequenceInputStream
{
public GzipCompressingInputStream(InputStream in) throws IOException
{
this(in,512);
}
public GzipCompressingInputStream(InputStream in, int bufferSize) throws IOException
{
super(new StatefullGzipStreamEnumerator(in,bufferSize));
}
static enum StreamState
{
HEADER,
CONTENT,
TRAILER
}
protected static class StatefullGzipStreamEnumerator implements Enumeration<InputStream>
{
protected final InputStream in;
protected final int bufferSize;
protected StreamState state;
public StatefullGzipStreamEnumerator(InputStream in, int bufferSize)
{
this.in=in;
this.bufferSize=bufferSize;
state=StreamState.HEADER;
}
public boolean hasMoreElements()
{
return state!=null;
}
public InputStream nextElement()
{
switch (state)
{
case HEADER:
state=StreamState.CONTENT;
return createHeaderStream();
case CONTENT:
state=StreamState.TRAILER;
return createContentStream();
case TRAILER:
state=null;
return createTrailerStream();
}
return null;
}
static final int GZIP_MAGIC = 0x8b1f;
static final byte[] GZIP_HEADER=new byte[] {
(byte) GZIP_MAGIC, // Magic number (short)
(byte)(GZIP_MAGIC >> 8), // Magic number (short)
Deflater.DEFLATED, // Compression method (CM)
0, // Flags (FLG)
0, // Modification time MTIME (int)
0, // Modification time MTIME (int)
0, // Modification time MTIME (int)
0, // Modification time MTIME (int)
0, // Extra flags (XFLG)
0 // Operating system (OS)
};
protected InputStream createHeaderStream()
{
return new ByteArrayInputStream(GZIP_HEADER);
}
protected InternalGzipCompressingInputStream contentStream;
protected InputStream createContentStream()
{
contentStream=new InternalGzipCompressingInputStream(new CRC32InputStream(in), bufferSize);
return contentStream;
}
protected InputStream createTrailerStream()
{
return new ByteArrayInputStream(contentStream.createTrailer());
}
}
/**
* Internal stream without header/trailer
*/
protected static class CRC32InputStream extends FilterInputStream
{
protected CRC32 crc = new CRC32();
protected long byteCount;
public CRC32InputStream(InputStream in)
{
super(in);
}
@Override
public int read() throws IOException
{
int val=super.read();
if (val>=0)
{
crc.update(val);
byteCount++;
}
return val;
}
@Override
public int read(byte[] b, int off, int len) throws IOException
{
len=super.read(b, off, len);
if (len>=0)
{
crc.update(b,off,len);
byteCount+=len;
}
return len;
}
public long getCrcValue()
{
return crc.getValue();
}
public long getByteCount()
{
return byteCount;
}
}
/**
* Internal stream without header/trailer
*/
protected static class InternalGzipCompressingInputStream extends DeflaterInputStream
{
protected final CRC32InputStream crcIn;
public InternalGzipCompressingInputStream(CRC32InputStream in, int bufferSize)
{
super(in, new Deflater(Deflater.DEFAULT_COMPRESSION, true),bufferSize);
crcIn=in;
}
public void close() throws IOException
{
if (in != null)
{
try
{
def.end();
in.close();
}
finally
{
in = null;
}
}
}
protected final static int TRAILER_SIZE = 8;
public byte[] createTrailer()
{
byte[] trailer= new byte[TRAILER_SIZE];
writeTrailer(trailer, 0);
return trailer;
}
/*
* Writes GZIP member trailer to a byte array, starting at a given
* offset.
*/
private void writeTrailer(byte[] buf, int offset)
{
writeInt((int)crcIn.getCrcValue(), buf, offset); // CRC-32 of uncompr. data
writeInt((int)crcIn.getByteCount(), buf, offset + 4); // Number of uncompr. bytes
}
/*
* Writes integer in Intel byte order to a byte array, starting at a
* given offset.
*/
private void writeInt(int i, byte[] buf, int offset)
{
writeShort(i & 0xffff, buf, offset);
writeShort((i >> 16) & 0xffff, buf, offset + 2);
}
/*
* Writes short integer in Intel byte order to a byte array, starting
* at a given offset
*/
private void writeShort(int s, byte[] buf, int offset)
{
buf[offset] = (byte)(s & 0xff);
buf[offset + 1] = (byte)((s >> 8) & 0xff);
}
}
}
import static org.junit.Assert.*;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.zip.CRC32;
import java.util.zip.GZIPInputStream;
import org.junit.Test;
public class TestGzipCompressingInputStream
{
@Test
public void test() throws Exception
{
testCompressor("test1 test2 test3");
testCompressor("1MB binary data",createTestPattern(1024*1024));
for (int i=0;i<4096;i++)
{
testCompressor(i+" bytes of binary data",createTestPattern(i));
}
}
protected byte[] createTestPattern(int size)
{
byte[] data=new byte[size];
byte pattern=0;
for (int i=0;i<size;i++)
{
data[i]=pattern++;
}
return data;
}
protected void testCompressor(String data) throws IOException
{
testCompressor("String: "+data,data.getBytes());
}
protected void testCompressor(String dataInfo, byte[] data) throws IOException
{
InputStream uncompressedIn=new ByteArrayInputStream(data);
InputStream compressedIn=new GzipCompressingInputStream(uncompressedIn);
InputStream uncompressedOut=new GZIPInputStream(compressedIn);
byte[] result=StreamHelper.readBinaryStream(uncompressedOut);
assertTrue("Test failed for: "+dataInfo,Arrays.equals(data,result));
}
}
如果你不想要的内容加载到大字节数组，需要真正的流媒体解决方案：
package x.y.z;
import org.apache.commons.io.IOUtils;
import java.io.*;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
import java.util.zip.ZipOutputStream;
/**
* Stream Compression Utility
*
* @author Thamme Gowda N
*/
public enum CompressionUtil {
INSTANCE;
public static final int NUM_THREADS = 5;
private final ExecutorService pool;
CompressionUtil(){
this.pool = Executors.newFixedThreadPool(NUM_THREADS);
}
public static CompressionUtil getInstance(){
return INSTANCE;
}
/**
* Supported compression type names
*/
public static enum CompressionType {
GZIP,
ZIP
}
/**
* Wraps the given stream in a Compressor stream based on given type
* @param sourceStream : Stream to be wrapped
* @param type : Compression type
* @return source stream wrapped in a compressor stream
* @throws IOException when some thing bad happens
*/
public static OutputStream getCompressionWrapper(OutputStream sourceStream,
CompressionType type) throws IOException {
switch (type) {
case GZIP:
return new GZIPOutputStream(sourceStream);
case ZIP:
return new ZipOutputStream(sourceStream);
default:
throw new IllegalArgumentException("Possible values :"
+ Arrays.toString(CompressionType.values()));
}
}
/**
* Gets Compressed Stream for given input Stream
* @param sourceStream : Input Stream to be compressed to
* @param type: Compression types such as GZIP
* @return Compressed Stream
* @throws IOException when some thing bad happens
*/
public static InputStream getCompressedStream(final InputStream sourceStream,
CompressionType type ) throws IOException {
if(sourceStream == null) {
throw new IllegalArgumentException("Source Stream cannot be NULL");
}
/**
* sourceStream --> zipperOutStream(->intermediateStream -)--> resultStream
*/
final PipedInputStream resultStream = new PipedInputStream();
final PipedOutputStream intermediateStream = new PipedOutputStream(resultStream);
final OutputStream zipperOutStream = getCompressionWrapper(intermediateStream, type);
Runnable copyTask = new Runnable() {
@Override
public void run() {
try {
int c;
while((c = sourceStream.read()) >= 0) {
zipperOutStream.write(c);
}
zipperOutStream.flush();
} catch (IOException e) {
IOUtils.closeQuietly(resultStream); // close it on error case only
throw new RuntimeException(e);
} finally {
// close source stream and intermediate streams
IOUtils.closeQuietly(sourceStream);
IOUtils.closeQuietly(zipperOutStream);
IOUtils.closeQuietly(intermediateStream);
}
}
};
getInstance().pool.submit(copyTask);
return resultStream;
}
public static void main(String[] args) throws IOException {
String input = "abcdefghij";
InputStream sourceStream = new ByteArrayInputStream(input.getBytes());
InputStream compressedStream =
getCompressedStream(sourceStream, CompressionType.GZIP);
GZIPInputStream decompressedStream = new GZIPInputStream(compressedStream);
List<String> lines = IOUtils.readLines(decompressedStream);
String output = lines.get(0);
System.out.println("test passed ? " + input.equals(output));
}
}
一个压缩输入流的工作实施例所用的流行的开源ESB找到骡 ： GZIPCompressorInputStream 。
它使用DeflaterInputStream由JRE用于压缩设置，预先考虑gzip的报头，并附加gzip的拖车（又名页脚）。
不幸的是，它是根据注册会计师执照 ，这似乎并不很常见。 此外，还有似乎没有单元测试。
看来我是晚了3年，但也许将是有用的人。 我的解决方案是类似于@迈克尔Wyraz的解决方案，唯一的区别就是我的解决方案是基于FilterInputStream
import java.io.ByteArrayInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.zip.CRC32;
import java.util.zip.Deflater;
public class GZipInputStreamDeflater extends FilterInputStream {
private static enum Stage {
HEADER,
DATA,
FINALIZATION,
TRAILER,
FINISH
}
private GZipInputStreamDeflater.Stage stage = Stage.HEADER;
private final Deflater deflater = new Deflater( Deflater.DEFLATED, true );
private final CRC32 crc = new CRC32();
/* GZIP header magic number */
private final static int GZIP_MAGIC = 0x8b1f;
private ByteArrayInputStream trailer = null;
private ByteArrayInputStream header = new ByteArrayInputStream( new byte[] {
(byte) GZIP_MAGIC, // Magic number (short)
(byte) ( GZIP_MAGIC >> 8 ), // Magic number (short)
Deflater.DEFLATED, // Compression method (CM)
0, // Flags (FLG)
0, // Modification time MTIME (int)
0, // Modification time MTIME (int)
0, // Modification time MTIME (int)
0, // Modification time MTIME (int)
0, // Extra flags (XFLG)
0, // Operating system (OS)
} );
public GZipInputStreamDeflater(InputStream in) {
super( in );
crc.reset();
}
@Override
public int read( byte[] b, int off, int len ) throws IOException {
int read = -1;
switch( stage ) {
case FINISH:
return -1;
case HEADER:
read = header.read( b, off, len );
if( header.available() == 0 ) {
stage = Stage.DATA;
}
return read;
case DATA:
byte[] b2 = new byte[len];
read = super.read( b2, 0, len );
if( read <= 0 ) {
stage = Stage.FINALIZATION;
deflater.finish();
return 0;
}
else {
deflater.setInput( b2, 0, read );
crc.update( b2, 0, read );
read = 0;
while( !deflater.needsInput() && len - read > 0 ) {
read += deflater.deflate( b, off + read, len - read, Deflater.NO_FLUSH );
}
return read;
}
case FINALIZATION:
if( deflater.finished() ) {
stage = Stage.TRAILER;
int crcVaue = (int) crc.getValue();
int totalIn = deflater.getTotalIn();
trailer = new ByteArrayInputStream( new byte[] {
(byte) ( crcVaue >> 0 ),
(byte) ( crcVaue >> 8 ),
(byte) ( crcVaue >> 16 ),
(byte) ( crcVaue >> 24 ),
(byte) ( totalIn >> 0 ),
(byte) ( totalIn >> 8 ),
(byte) ( totalIn >> 16 ),
(byte) ( totalIn >> 24 ),
} );
return 0;
}
else {
read = deflater.deflate( b, off, len, Deflater.FULL_FLUSH );
return read;
}
case TRAILER:
read = trailer.read( b, off, len );
if( trailer.available() == 0 ) {
stage = Stage.FINISH;
}
return read;
}
return -1;
}
@Override
public void close( ) throws IOException {
super.close();
deflater.end();
if( trailer != null ) {
trailer.close();
}
header.close();
}
}
用法：
AmazonS3Client s3client = new AmazonS3Client( ... );
try ( InputStream in = new GZipInputStreamDeflater( new URL( "http://....../very-big-file.csv" ).openStream() ); ) {
PutObjectRequest putRequest = new PutObjectRequest( "BUCKET-NAME", "/object/key", in, new ObjectMetadata() );
s3client.putObject( putRequest );
}
要压缩数据，您需要的GZIPOutputStream 。 但既然你需要读取数据回，就好像从InputStream需要OutputStream的转换为一个InputStream。 您可以使用的getBytes（）这样做：
GZIPOutputStream gout = new GZIPOutputStream(out);
//... Code to read from your original uncompressed data and write to out.
//Convert to InputStream.
new ByteArrayInputStream(gout.getBytes());
但这种方法有，你需要先阅读中的所有数据的限制 - 这意味着你必须有足够的内存来保存缓冲区。
-用管道替代方法是在这个线程提到如何转换的OutputStream到的InputStream？
你不应该在看GZIPOutputStream在这种情况下？
public OutputStream getCompressedStream(InputStream input) {
OutputStream output = new GZIPOutputStream(new ByteArrayOutputStream());
IOUtils.copy(input, output);
return output;
}
没有DeflatingGZIPInputStream在JRE。 要使用“deflate”压缩格式缩小，使用java.util.zip.DeflaterInputStream和java.util.zip.DeflaterOutputStream ：
public InputStream getCompressedStream(InputStream unCompressedStream) {
return new DeflaterInputStream(unCompressedStream);
}
你可以从派生类java.util.zip.DeflaterInputStream通过查看的源gzip格式的放气java.util.zip.GZIPOutputStream 。
您可以使用EasyStream 。
try(final InputStreamFromOutputStream<Void> isOs = new InputStreamFromOutputStream<Void>() {
@Override
protected void produce(final OutputStream dataSink) throws Exception {
InputStream in = new GZIPInputStream(unCompressedStream);
IOUtils.copy(in, dataSink);
}
}) {
//You can use the compressed input stream here
} catch (final IOException e) {
//Handle exceptions here
}
的PipedOutputStream让你写一个GZIPOutputStream，并通过一个InputStream公开这些数据。 它有一个固定的存储器成本，不同于数据的整个位流缓存器到阵列或文件的其他解决方案。 唯一的问题是你不能阅读和在同一个线程写，你必须使用一个单独的一个。
private InputStream gzipInputStream(InputStream in) throws IOException {
PipedInputStream zipped = new PipedInputStream();
PipedOutputStream pipe = new PipedOutputStream(zipped);
new Thread(
() -> {
try(OutputStream zipper = new GZIPOutputStream(pipe)){
IOUtils.copy(in, zipper);
} catch (IOException e) {
e.printStackTrace();
}
}
).start();
return zipped;
}
我会建议使用GzipCompressorInputStream从Apache的百科全书压缩 。
{var%20f='http://v.t.sina.com.cn/share/share.php?appkey=1515056452',u=z||d.location,p=['&url=',e(u),'&title=',e(t||d.title),'&source=',e(r),'&sourceUrl=',e(l),'&content=',c||'gb2312','&pic=',e(p||'')].join('');function%20a(){if(!window.open([f,p].join(''),'mb',['toolbar=0,status=0,resizable=1,width=440,height=430,left=',(s.width-440)/2,',top=',(s.height-430)/2].join('')))u.href=[f,p].join('');};if(/Firefox/.test(navigator.userAgent))setTimeout(a,0);else%20a();})(screen,document,encodeURIComponent,'','','https://img.manongdao.com/sparkler-677774__340.jpg', '推荐 三岁会撩人 的文章《压缩用gzip的InputStream(Compress an InputStream with g》','https://www.manongdao.com/article-1021696.html','页面编码gb2312|utf-8默认gb2312'));){kind=link}