Gzip: Difference between revisions
Jump to navigation
Jump to search
imported>Connoisseurship →File format: format <code> |
imported>Fgnievinski |
||
| Line 1: | Line 1: | ||
{{short description| | {{short description|File format and file compression program}} | ||
{{Use dmy dates|date=June 2023}} | {{Use dmy dates|date=June 2023}} | ||
{{Lowercase title}} | {{Lowercase title}} | ||
| Line 5: | Line 5: | ||
| name = gzip (software) | | name = gzip (software) | ||
| logo = Gzip-Logo.svg | | logo = Gzip-Logo.svg | ||
| author = {{ | | author = {{ubl|[[Jean-Loup Gailly]]|[[Mark Adler]]}} | ||
}} | |||
| developer = [[GNU Project]] | | developer = [[GNU Project]] | ||
| released = {{Start date and age|df=yes|1992|10|31}} | | released = {{Start date and age|df=yes|1992|10|31}} | ||
| Line 19: | Line 16: | ||
| programming language = [[C (programming language)|C]] | | programming language = [[C (programming language)|C]] | ||
| license = [[GNU General Public License|GPL-3.0-or-later]] | | license = [[GNU General Public License|GPL-3.0-or-later]] | ||
| website = {{URL|//www.gnu.org/software/gzip/}} | | website = | ||
| repo = {{URL|// | {{URL|https://www.gnu.org/software/gzip/|gnu.org/software/gzip}}<ref>{{cite journal |journal=IEEE Access |doi=10.1109/ACCESS.2020.3041688 |doi-access=free|title=On the Potential of V2X Message Compression for Vehicular Networks |page=14 |postscript=Reference number [20] "GNU Documentation: Gzip"}}</ref><br> | ||
{{URL|https://gzip.org}} (obsolete) | |||
| repo = {{URL|https://savannah.gnu.org/projects/gzip}}<ref>{{cite web |title=Gzip - GNU Project |url=https://www.gnu.org/software/gzip/ |archive-url=https://web.archive.org/web/20250505101843/https://www.gnu.org/software/gzip/ |archive-date=5 May 2025 |quote=For development sources and other information, please see the Gzip project page at savannah.gnu.org.}}</ref> | |||
}} | }} | ||
'''gzip''' is a [[file format]] and a [[ | '''gzip''' is a [[file format]] and a [[file compression]] [[computer program|program]]. The program uses the [[Deflate]] algorithm to compress and decompress a single file using the [[#File format|gzip file format]]. | ||
gzip was released in 1992 as a [[free software]] replacement for the ''[[compress]]'' program because its compression algorithm, [[LZW]], was covered by [[patent]]s from [[Unisys]] and [[IBM]],<ref>{{cite web |title=Gzip - GNU Project |url=https://www.gnu.org/software/gzip/ |archive-url=https://web.archive.org/web/20250505101843/https://www.gnu.org/software/gzip/ |archive-date=5 May 2025 |quote=We developed this program as a replacement for compress because of the Unisys and IBM patents covering the LZW algorithm used by compress.}}</ref> which did not expire until 2003 and 2004.<ref>{{cite web |title=Unisys {{!}} LZW Patent and Software Information |url=http://www.unisys.com/about__unisys/lzw/ |archive-url=https://web.archive.org/web/20090626052026/http://www.unisys.com/about__unisys/lzw/ |archive-date=26 June 2009 |quote=Unisys U.S. LZW Patent No. 4,558,302 expired on June 20, 2003, the counterpart patents in the United Kingdom, France, Germany and Italy expired on June 18, 2004, the Japanese counterpart patents expired on June 20, 2004 and the counterpart Canadian patent expired on July 7, 2004.}}</ref> [[Jean-Loup Gailly]]<!--Person prefers lowercase l (ell), see his article--> designed the gzip file format,<ref>{{Cite IETF|rfc=1952|title=Acknowledgements|page=10|quote=Jean-Loup Gailly designed the gzip format and ...}}</ref> which was later specified by RFC 1952,<ref>{{cite IETF |rfc=1952 |title=GZIP file format specification version 4.3}}</ref> and originally wrote the gzip program. [[Mark Adler]] wrote the decompression part. gzip is now developed by the [[GNU]] project.<ref>{{cite web |title=Gzip - GNU Project |url=https://www.gnu.org/software/gzip/ |archive-url=https://web.archive.org/web/20250505101843/https://www.gnu.org/software/gzip/ |archive-date=5 May 2025 |quote=GNU Gzip is a popular data compression program originally written by Jean-Loup Gailly for the GNU project. Mark Adler wrote the decompression part.}}</ref> | |||
As the file format can be decompressed via a [[streaming algorithm]], it is commonly used in stream-based technology such as [[HTTP compression|Web protocols]], [[Electronic data interchange|data interchange]] and [[Extract, transform, load|ETL]] (in [[Standard streams|standard pipes]]). | As the file format can be decompressed via a [[streaming algorithm]], it is commonly used in stream-based technology such as [[HTTP compression|Web protocols]], [[Electronic data interchange|data interchange]] and [[Extract, transform, load|ETL]] (in [[Standard streams|standard pipes]]). | ||
| Line 32: | Line 33: | ||
| extension = <code>.gz</code>, <code>.tgz</code>, <code>.gzip</code> | | extension = <code>.gz</code>, <code>.tgz</code>, <code>.gzip</code> | ||
| mime = <code>application/gzip</code><ref>{{cite IETF|title=The 'application/zlib' and 'application/gzip' Media Types|rfc=6713|publisher=Internet Engineering Task Force|access-date=2014-03-01}}</ref> | | mime = <code>application/gzip</code><ref>{{cite IETF|title=The 'application/zlib' and 'application/gzip' Media Types|rfc=6713|publisher=Internet Engineering Task Force|access-date=2014-03-01}}</ref> | ||
| magic = <code>1F 8B</code> | | magic = {{ubl|<code>1F 8B</code> (specification)|<code>1F 8B 08</code> (de facto)}} | ||
| | | developer = [[Jean-Loup Gailly]] | ||
| | | type = [[Data compression]] | ||
| | | compression = [[Lossless]] | ||
| | | container for = [[Deflate stream]] | ||
| standard = | | standard = {{IETF RFC|1952}} | ||
| uniform_type = <code>org.gnu.gnu-zip-archive</code> | | uniform_type = {{ubl|<code>org.gnu.gnu-zip-archive</code>|<code>org.gnu.gnu-zip-tar-archive</code>}} | ||
| max_size = Unlimited | | max_size = Unlimited | ||
}} | }} | ||
[[File:Targzip.svg|thumb|left|gzip can be combined with the [[tar (file format)|tar]] program to compress multiple files.]] | |||
gzip | |||
[[ | A gzip file (described in the table below) consists of a 10-[[byte]] [[Header (computing)|header]], optional extra header fields, [[Deflate]]-compressed data, and an 8-byte [[Trailer (computing)|trailer]]. | ||
Although multiple streams may be [[Concatenation|concatenated]] (gzipped files are simply decompressed concatenated as if they were originally one file),<ref>{{cite web| title=GNU Gzip: Advanced usage| url=https://www.gnu.org/software/gzip/manual/html_node/Advanced-usage.html| website=Gnu.org| access-date=2012-11-28| archive-date=24 December 2012| archive-url=https://web.archive.org/web/20121224211417/https://www.gnu.org/software/gzip/manual/html_node/Advanced-usage.html| url-status=live}}</ref> normally only a single file is compressed.<ref>{{cite web| title=Can gzip compress several files into a single archive?| url=http://www.gzip.org/#faq16|website=Gnu.org| access-date=2010-01-27 |archive-url=https://web.archive.org/web/20100722004012/http://www.gzip.org/#faq16 |archive-date=2010-07-22}}</ref> Compressed archives are typically created by assembling collections of files into a single [[tar (file format)|tar]] archive (also called [[tar (file format)|tarball]]),<ref>{{cite web|title=tarball, The Jargon File, version 4.4.7|url=http://www.catb.org/jargon/html/T/tarball.html|website=Catb.org|access-date=2010-01-27|archive-date=20 March 2017|archive-url=https://web.archive.org/web/20170320183848/http://www.catb.org/jargon/html/T/tarball.html|url-status=live}}</ref> and then compressing that archive with gzip. The final compressed file usually has the extension <code>.tar.gz</code> or <code>. | Although multiple streams may be [[Concatenation|concatenated]] (gzipped files are simply decompressed concatenated as if they were originally one file),<ref>{{cite web| title=GNU Gzip: Advanced usage| url=https://www.gnu.org/software/gzip/manual/html_node/Advanced-usage.html| website=Gnu.org| access-date=2012-11-28| archive-date=24 December 2012| archive-url=https://web.archive.org/web/20121224211417/https://www.gnu.org/software/gzip/manual/html_node/Advanced-usage.html| url-status=live}}</ref> normally only a single file is compressed.<ref>{{cite web| title=Can gzip compress several files into a single archive?| url=http://www.gzip.org/#faq16|website=Gnu.org| access-date=2010-01-27 |archive-url=https://web.archive.org/web/20100722004012/http://www.gzip.org/#faq16 |archive-date=2010-07-22}}</ref> Compressed archives are typically created by assembling collections of files into a single [[tar (file format)|tar]] archive (also called [[tar (file format)|tarball]]),<ref>{{cite web|title=tarball, The Jargon File, version 4.4.7|url=http://www.catb.org/jargon/html/T/tarball.html|website=Catb.org|access-date=2010-01-27|archive-date=20 March 2017|archive-url=https://web.archive.org/web/20170320183848/http://www.catb.org/jargon/html/T/tarball.html|url-status=live}}</ref> and then compressing that archive with gzip. The final compressed file usually has the extension <code>.tar.gz</code>, <code>.tgz</code>, <code>.gz</code>, or <code>.gzip</code>. | ||
gzip is not to be confused with the [[ZIP (file format)|ZIP]] archive format, which also uses [[Deflate|DEFLATE]]. The ZIP format can hold collections of files without an external archiver, but is less compact than compressed [[tar (computing)|tarballs]] holding the same data, because it compresses files individually and cannot take advantage of redundancy between files ([[solid compression]]). | gzip is not to be confused with the [[ZIP (file format)|ZIP]] archive format, which also uses [[Deflate|DEFLATE]]. The ZIP format can hold collections of files without an external archiver, but is less compact than compressed [[tar (computing)|tarballs]] holding the same data, because it compresses files individually and cannot take advantage of redundancy between files ([[solid compression]]). | ||
| Line 54: | Line 53: | ||
=== File structure === | === File structure === | ||
[[Endianness]] is ''[[little-endian]]''.<ref>{{Cite IETF|rfc=1952|title=Overall conventions|page=4|quote=All multi-byte numbers in the format described here are stored with the least-significant byte first (at the lower memory address).}}</ref> | |||
{| class="wikitable" | {| class="wikitable" | ||
| | ![[Offset (computer science)|Offset]] (bytes) | ||
!Field<ref>{{Cite IETF|rfc=1952|title=Member format|page=5}}</ref> | |||
!Size (bytes) | |||
!Size | |||
!Description | !Description | ||
|- | |- | ||
| Line 134: | Line 132: | ||
|0 or <code>XLEN</code> | |0 or <code>XLEN</code> | ||
|- | |- | ||
|{{varies}} | |rowspan=6" {{varies}} | ||
|File name | |File name | ||
|{{varies|0 or varies}} | |{{varies|0 or varies}} | ||
|[[Null-terminated string|Null-terminated]] '''name of the file being compressed'''. Present if the '''FNAME''' flag is set. Encoded as [[ISO/IEC 8859-1|ISO 8859-1]] (<code>latin-1</code>). Converted to [[lowercase]] on [[case-insensitive]] filesystems. Empty if the compressed data did not come from a named file. | |[[Null-terminated string|Null-terminated]] '''name of the file being compressed'''. Present if the '''FNAME''' flag is set. Encoded as [[ISO/IEC 8859-1|ISO 8859-1]] (<code>latin-1</code>). Converted to [[lowercase]] on [[case-insensitive]] filesystems. Empty if the compressed data did not come from a named file. | ||
|- | |- | ||
|File comment | |File comment | ||
|{{varies|0 or varies}} | |{{varies|0 or varies}} | ||
|Null-terminated '''file comment''' intended for human consumption. Present if the '''FCOMMENT''' flag is set. Encoded as ISO 8859-1 (<code>latin-1</code>). [[Newline|Newlines]] should use a single line feed (LF) character. | |Null-terminated '''file comment''' intended for human consumption. Present if the '''FCOMMENT''' flag is set. Encoded as ISO 8859-1 (<code>latin-1</code>). [[Newline|Newlines]] should use a single line feed (LF) character. | ||
|- | |- | ||
|CRC16 | |CRC16 | ||
|0 or 2 | |0 or 2 | ||
|Two least significant bytes of the [[CRC-32]] (ISO 3309) of all bytes in the gzip file up to (not including) this field. Present if the '''FHCRC''' flag is set. | |Two least significant bytes of the [[CRC-32]] (ISO 3309) of all bytes in the gzip file up to (not including) this field. Present if the '''FHCRC''' flag is set. | ||
|- | |- | ||
|Compressed data | |Compressed data | ||
|{{varies}} | |{{varies}} | ||
| | |[[Deflate stream]]. | ||
|- | |- | ||
|CRC32 | |CRC32 | ||
|4 | |4 | ||
|'''CRC-32''' (ISO 3309) of the '''uncompressed data'''. | |'''CRC-32''' (ISO 3309) of the '''uncompressed data'''. | ||
|- | |- | ||
|ISIZE | |ISIZE | ||
|4 | |4 | ||
| Line 167: | Line 160: | ||
==Implementations== | ==Implementations== | ||
{{Infobox software | {{Infobox software | ||
| name | | name = NetBSD Gzip / FreeBSD Gzip | ||
| developer | | developer = The NetBSD Foundation | ||
| genre | | genre = [[Data compression]] | ||
| programming language = [[C (programming language)|C]] | | programming language = [[C (programming language)|C]] | ||
| license | | license = [[BSD licenses|Simplified BSD License]] | ||
| website | | website = hide <!-- Don't display the GNU Gzip URL from Wikidata on the BSD Gzip infobox --> | ||
| repo | | repo = {{URL|//cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/gzip/}} | ||
}} | }} | ||
Various implementations of the program have been written. The most commonly known is the GNU Project's implementation using [[LZ77|Lempel-Ziv]] coding (LZ77). [[OpenBSD]]'s version of gzip is actually the [[compress]] program, to which support for the gzip format was added in OpenBSD 3.4. The "g" in this specific version stands for ''[[wikt:gratis|gratis]]''.<ref>{{cite web|url=https://man.openbsd.org/gzip.1#HISTORY|title=OpenBSD gzip(1) manual page|website=Openbsd.org|access-date=2018-02-04|publisher=OpenBSD}}</ref> [[FreeBSD]], [[DragonFly BSD]] and [[NetBSD]] use a BSD-licensed implementation instead of the GNU version; it is actually a [[command-line interface]] for [[zlib]] intended to be compatible with the GNU implementations' options.<ref>{{cite web|url=http://man.freebsd.org/gzip|title=gzip|publisher=Man.freebsd.org|date=2011-10-09|access-date=2014-03-01|archive-date=17 December 2019|archive-url=https://web.archive.org/web/20191217104949/https://www.freebsd.org/cgi/man.cgi?gzip|url-status=live}}</ref> These implementations originally come from [[NetBSD]], and support decompression of [[bzip2]] and the Unix [[pack (compression)|pack]] format. | Various implementations of the program have been written. The most commonly known is the GNU Project's implementation using [[LZ77|Lempel-Ziv]] coding (LZ77). [[OpenBSD]]'s version of gzip is actually the [[compress]] program, to which support for the gzip format was added in OpenBSD 3.4. The "g" in this specific version stands for ''[[wikt:gratis|gratis]]''.<ref>{{cite web|url=https://man.openbsd.org/gzip.1#HISTORY|title=OpenBSD gzip(1) manual page|website=Openbsd.org|access-date=2018-02-04|publisher=OpenBSD}}</ref> [[FreeBSD]], [[DragonFly BSD]] and [[NetBSD]] use a BSD-licensed implementation instead of the GNU version; it is actually a [[command-line interface]] for [[zlib]] intended to be compatible with the GNU implementations' options.<ref>{{cite web|url=http://man.freebsd.org/gzip|title=gzip|publisher=Man.freebsd.org|date=2011-10-09|access-date=2014-03-01|archive-date=17 December 2019|archive-url=https://web.archive.org/web/20191217104949/https://www.freebsd.org/cgi/man.cgi?gzip|url-status=live}}</ref> These implementations originally come from [[NetBSD]], and support decompression of [[bzip2]] and the Unix [[pack (compression)|pack]] format. | ||
| Line 182: | Line 175: | ||
=== Damage recovery === | === Damage recovery === | ||
Data in blocks prior to the first damaged part of the archive is usually fully readable. Data from blocks not demolished by damage that are located afterward ''may'' be [[Data recovery|recoverable]] through difficult workarounds.<ref>[https://web.archive.org/web/20180708075208/http://www.gzip.org/recover.txt Recovering a damaged .gz file – Jean- | Data in blocks prior to the first damaged part of the archive is usually fully readable. Data from blocks not demolished by damage that are located afterward ''may'' be [[Data recovery|recoverable]] through difficult workarounds.<ref>[https://web.archive.org/web/20180708075208/http://www.gzip.org/recover.txt Recovering a damaged .gz file – Jean-Loup Gailly, GZip.org]</ref> | ||
== Derivatives and other uses == | == Derivatives and other uses == | ||
The [[tar (computing)|tar]] utility included in most Linux distributions can extract .tar.gz files by passing the {{mono|z}} option, e.g., {{mono|tar -zxf file.tar.gz}}, where <code>-z</code> instructs decompression, <code>-x</code> means extraction, and <code>-f</code> specifies the name of the compressed archive file to extract from. Optionally, <code>-v</code> (''verbose'') lists files as they are being extracted.<ref>{{cite web |title=How To Extract / Unzip tar.gz Files From Linux Command Line |url=https://phoenixnap.com/kb/extract-tar-gz-files-linux-command-line |website=Knowledge Base by phoenixNAP |access-date=12 January 2022 |date=14 November 2019}}</ref> | The [[tar (computing)|tar]] utility included in most Linux distributions can extract .tar.gz files by passing the {{mono|z}} option, e.g., {{mono|tar -zxf file.tar.gz}}, where <code>-z</code> instructs decompression, <code>-x</code> means extraction, and <code>-f</code> specifies the name of the compressed archive file to extract from. Optionally, <code>-v</code> (''verbose'') lists files as they are being extracted.<ref>{{cite web |title=How To Extract / Unzip tar.gz Files From Linux Command Line |url=https://phoenixnap.com/kb/extract-tar-gz-files-linux-command-line |website=Knowledge Base by phoenixNAP |access-date=12 January 2022 |date=14 November 2019}}</ref> | ||
[[zlib]] | The [[zlib]] library supports the gzip file format.<ref>{{cite web |title=zlib 1.3.1 Manual |url=https://zlib.net/manual.html |archive-url=https://web.archive.org/web/20240208162415/https://zlib.net/manual.html |access-date=8 February 2024 |archive-date=8 February 2024 |at=Introduction |quote=The library also supports reading and writing files in gzip (.gz) format ...}}</ref> | ||
The gzip format is used in [[HTTP compression]], a technique used to speed up the sending of [[HTML]] and other content on the [[World Wide Web]]. It is one of the three standard formats for HTTP compression as specified in RFC 2616. This [[Request for Comments|RFC]] also specifies a zlib format (called "DEFLATE"), which is equal to the gzip format except that gzip adds eleven bytes of overhead in the form of headers and trailers. Still, the gzip format is sometimes recommended over zlib because [[Internet Explorer]] does not implement the standard correctly and cannot handle the zlib format as specified in RFC 1950.<ref>{{cite web |first=Eric |last=Lawrence |title=Compressing the Web |url=http://blogs.msdn.com/b/ieinternals/archive/2014/10/21/http-compression-optimize-file-formats-with-deflate.aspx |date=21 November 2014 |website=MSDN Blogs > IEInternals |publisher=[[Microsoft]] |access-date=2 November 2015 |archive-date=28 October 2015 |archive-url=https://web.archive.org/web/20151028172035/http://blogs.msdn.com/b/ieinternals/archive/2014/10/21/http-compression-optimize-file-formats-with-deflate.aspx |url-status=live }}</ref> | The gzip format is used in [[HTTP compression]], a technique used to speed up the sending of [[HTML]] and other content on the [[World Wide Web]]. It is one of the three standard formats for HTTP compression as specified in RFC 2616. This [[Request for Comments|RFC]] also specifies a zlib format (called "DEFLATE"), which is equal to the gzip format except that gzip adds eleven bytes of overhead in the form of headers and trailers. Still, the gzip format is sometimes recommended over zlib because [[Internet Explorer]] does not implement the standard correctly and cannot handle the zlib format as specified in RFC 1950.<ref>{{cite web |first=Eric |last=Lawrence |title=Compressing the Web |url=http://blogs.msdn.com/b/ieinternals/archive/2014/10/21/http-compression-optimize-file-formats-with-deflate.aspx |date=21 November 2014 |website=MSDN Blogs > IEInternals |publisher=[[Microsoft]] |access-date=2 November 2015 |archive-date=28 October 2015 |archive-url=https://web.archive.org/web/20151028172035/http://blogs.msdn.com/b/ieinternals/archive/2014/10/21/http-compression-optimize-file-formats-with-deflate.aspx |url-status=live }}</ref> | ||
Since the late 1990s, [[bzip2]], a file compression utility based on a block-sorting algorithm, has gained some popularity as a gzip replacement. It produces considerably smaller files (especially for source code and other structured text), but at the cost of memory and processing time (up to a factor of 4).<ref name="benchmark">{{cite web |url=http://compressionratings.com/comp.cgi?7-zip+9.12b++bzip2+1.0.5++gzip+1.3.3+-5 |title=Comparison Tool: 7-zip vs bzip2 vs gzip | website=compressionratings.com |access-date=1 November 2014 |url-status=dead |archive-url=https://web.archive.org/web/20141101212919/http://compressionratings.com/comp.cgi?7-zip+9.12b++bzip2+1.0.5++gzip+1.3.3+-5 |archive-date=1 November 2014 }}</ref> | Since the late 1990s, [[bzip2]], a file compression utility based on a block-sorting algorithm, has gained some popularity as a gzip replacement. It produces considerably smaller files (especially for source code and other structured text), but at the cost of memory and processing time (up to a factor of 4).<ref name="benchmark">{{cite web |url=http://compressionratings.com/comp.cgi?7-zip+9.12b++bzip2+1.0.5++gzip+1.3.3+-5 |title=Comparison Tool: 7-zip vs bzip2 vs gzip | website=compressionratings.com |access-date=1 November 2014 |url-status=dead |archive-url=https://web.archive.org/web/20141101212919/http://compressionratings.com/comp.cgi?7-zip+9.12b++bzip2+1.0.5++gzip+1.3.3+-5 |archive-date=1 November 2014 }}</ref> | ||
| Line 202: | Line 193: | ||
{{Portal|Free and open-source software}} | {{Portal|Free and open-source software}} | ||
* {{Annotated link|Brotli}} | * {{Annotated link|Brotli}} | ||
* {{Annotated link|Comparison of file archivers}} | * {{Annotated link|Comparison of file archivers}} | ||
* {{Annotated link|List of archive formats}} | * {{Annotated link|List of archive formats}} | ||
| Line 209: | Line 199: | ||
* {{Annotated link|zlib}} | * {{Annotated link|zlib}} | ||
== | ==References== | ||
{{Reflist}} | {{Reflist}} | ||
==External links== | ==External links== | ||
* {{Official website|//gnu.org/software/gzip/}} | * {{Official website|https://www.gnu.org/software/gzip/}} | ||
{{Archive formats}} | {{Archive formats}} | ||
| Line 234: | Line 219: | ||
[[Category:Plan 9 commands]] | [[Category:Plan 9 commands]] | ||
[[Category:Unix archivers and compression-related utilities]] | [[Category:Unix archivers and compression-related utilities]] | ||
[[Category:Software using the GNU General Public License]] | |||