Gzip

gzip (software)

File:Gzip-Logo.svg
Original author(s)	Jean-Loup Gailly Mark Adler
Developer(s)	GNU Project
Initial release	31 October 1992 (1992-10-31)
Repository	savannah.gnu.org/projects/gzip
Written in	C
Operating system	Unix-like, Plan 9, Inferno
Type	Data compression
License	GPL-3.0-or-later
Website	gnu.org/software/gzip gzip.org (obsolete)

gzip is a file format and a file compression program. The program uses the Deflate algorithm to compress and decompress a single file using the gzip file format.

gzip was released in 1992 as a free software replacement for the compress program because its compression algorithm, LZW, was covered by patents from Unisys and IBM, which did not expire until 2003 and 2004. Jean-Loup Gailly designed the gzip file format, which was later specified by RFC 1952, and originally wrote the gzip program. Mark Adler wrote the decompression part. gzip is now developed by the GNU project.

As the file format can be decompressed via a streaming algorithm, it is commonly used in stream-based technology such as Web protocols, data interchange and ETL (in standard pipes).

File format

gzip (file format)

Filename extension	.gz, .tgz, .gzip
Internet media type	application/gzip
Uniform Type Identifier (UTI)	org.gnu.gnu-zip-archive org.gnu.gnu-zip-tar-archive
Magic number	1F 8B (specification) 1F 8B 08 (de facto)
Developed by	Jean-Loup Gailly
Type of format	Data compression
Container for	Deflate stream
Standard	RFC 1952

File:Targzip.svg

gzip can be combined with the tar program to compress multiple files.

A gzip file (described in the table below) consists of a 10-byte header, optional extra header fields, Deflate-compressed data, and an 8-byte trailer.

Although multiple streams may be concatenated (gzipped files are simply decompressed concatenated as if they were originally one file), normally only a single file is compressed. Compressed archives are typically created by assembling collections of files into a single tar archive (also called tarball), and then compressing that archive with gzip. The final compressed file usually has the extension .tar.gz, .tgz, .gz, or .gzip.

gzip is not to be confused with the ZIP archive format, which also uses DEFLATE. The ZIP format can hold collections of files without an external archiver, but is less compact than compressed tarballs holding the same data, because it compresses files individually and cannot take advantage of redundancy between files (solid compression). The gzip file format is also not to be confused with that of the compress utility, based on LZW, with extension .Z; however, the gunzip utility is able to decompress .Z files.

File structure

Endianness is little-endian.

Offset (bytes)	Field	Size (bytes)	Description
0	ID1	1	Magic number. Must be 1F 8B.
1	ID2	1
2	CM	1	Compression method. Must be 8 (Deflate).
3	FLG	1	Flags. Reserved bits must be zero. Bit 0 (LSb): FTEXT. Set by the compressor to indicate the file encoding is probably ASCII. Bit 1: FHCRC Bit 2: FEXTRA Bit 3: FNAME Bit 4: FCOMMENT Bit 5: Reserved Bit 6: Reserved Bit 7 (MSb): Reserved
4	MTIME	4	Unix time when the file was last modified. If the compressed data did not come from a file, MTIME is the Unix time when compression started. 0 means no timestamp is available.
8	XFL	1	Extra flags. Deflate-specific flags. 0: None (default value) 2: Best compression (level 9) 4: Fastest compression (level 1)
9	OS	1	Filesystem on which compression occurred. 255: Unknown (default value) 0: FAT filesystem (MS-DOS, OS/2, NT/Win32) 1: Amiga 2: OpenVMS (or VMS) 3: Unix 4: VM/CMS 5: Atari TOS 6: HPFS filesystem (OS/2, NT) 7: Macintosh 8: Z-System 9: CP/M 10: TOPS-20 11: NTFS filesystem (NT) 12: QDOS 13: Acorn RISCOS
10	XLEN	0 or 2	Extra field is a sequence of subfields. XLEN is the size in bytes of the extra field. Both are present if the FEXTRA flag is set. Each subfield starts with SI1 SI2 (a two-byte identifier; typically two ASCII letters with some mnemonic value) followed by a two-byte LEN value indicating the remaining number of bytes in the subfield. Subfield IDs with SI2 = 0 are reserved for future use.
12	Extra field	0 or XLEN
Varies	File name	0 or varies	Null-terminated name of the file being compressed. Present if the FNAME flag is set. Encoded as ISO 8859-1 (latin-1). Converted to lowercase on case-insensitive filesystems. Empty if the compressed data did not come from a named file.
File comment	0 or varies	Null-terminated file comment intended for human consumption. Present if the FCOMMENT flag is set. Encoded as ISO 8859-1 (latin-1). Newlines should use a single line feed (LF) character.
CRC16	0 or 2	Two least significant bytes of the CRC-32 (ISO 3309) of all bytes in the gzip file up to (not including) this field. Present if the FHCRC flag is set.
Compressed data	Varies	Deflate stream.
CRC32	4	CRC-32 (ISO 3309) of the uncompressed data.
ISIZE	4	Size (in bytes) of the uncompressed data modulo Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle 2^{32}} .

Implementations

NetBSD Gzip / FreeBSD Gzip

Developer(s)	The NetBSD Foundation
Repository	cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/gzip/
Written in	C
Type	Data compression
License	Simplified BSD License

Various implementations of the program have been written. The most commonly known is the GNU Project's implementation using Lempel-Ziv coding (LZ77). OpenBSD's version of gzip is actually the compress program, to which support for the gzip format was added in OpenBSD 3.4. The "g" in this specific version stands for gratis. FreeBSD, DragonFly BSD and NetBSD use a BSD-licensed implementation instead of the GNU version; it is actually a command-line interface for zlib intended to be compatible with the GNU implementations' options. These implementations originally come from NetBSD, and support decompression of bzip2 and the Unix pack format.

An alternative compression program achieving 3-8% better compression is Zopfli. It achieves gzip-compatible compression using more exhaustive algorithms, at the expense of compression time required. It does not affect decompression time.

pigz, written by Mark Adler, is compatible with gzip and speeds up compression by using all available CPU cores and threads.

Damage recovery

Data in blocks prior to the first damaged part of the archive is usually fully readable. Data from blocks not demolished by damage that are located afterward may be recoverable through difficult workarounds.

Derivatives and other uses

The tar utility included in most Linux distributions can extract .tar.gz files by passing the z option, e.g., tar -zxf file.tar.gz, where -z instructs decompression, -x means extraction, and -f specifies the name of the compressed archive file to extract from. Optionally, -v (verbose) lists files as they are being extracted.

The zlib library supports the gzip file format.

The gzip format is used in HTTP compression, a technique used to speed up the sending of HTML and other content on the World Wide Web. It is one of the three standard formats for HTTP compression as specified in RFC 2616. This RFC also specifies a zlib format (called "DEFLATE"), which is equal to the gzip format except that gzip adds eleven bytes of overhead in the form of headers and trailers. Still, the gzip format is sometimes recommended over zlib because Internet Explorer does not implement the standard correctly and cannot handle the zlib format as specified in RFC 1950.

Since the late 1990s, bzip2, a file compression utility based on a block-sorting algorithm, has gained some popularity as a gzip replacement. It produces considerably smaller files (especially for source code and other structured text), but at the cost of memory and processing time (up to a factor of 4).

AdvanceCOMP, Zopfli, libdeflate and 7-Zip can produce gzip-compatible files, using an internal DEFLATE implementation with better compression ratios than gzip itself—at the cost of more processor time compared to the reference implementation.

Research published in 2023 showed that simple lossless compression techniques such as gzip could be combined with a k-nearest-neighbor classifier to create an attractive alternative to deep neural networks for text classification in natural language processing. This approach has been shown to equal and in some cases outperform conventional approaches such as BERT due to low resource requirements, e.g. no requirement for GPU hardware.

See also

• Brotli
• Comparison of file archivers
• List of archive formats
• List of POSIX commands
• Open file format
• zlib – Data format and compression library

External links

• Official website