Editing Encyclopedia:Database download (section)

==Dealing with large files==
As files grow in size, so does the likelihood they will exceed some limit of a computing device. Each operating system, file system, hard storage device, and software (application) has a maximum file size limit. Each one of these will likely have a different maximum, and the lowest limit of all of them will become the file size limit for a storage device.

The older the software in a computing device, the more likely it will have a 2 GB file limit somewhere in the system. This is due to older software using 32-bit integers for file indexing, which limits file sizes to 2^31 bytes (2 GB) (for signed integers), or 2^32 (4 GB) (for unsigned integers). Older [[C (programming language)|C]] [[library (computing)|programming libraries]] have this 2 or 4 GB limit, but the newer file libraries have been converted to 64-bit integers thus supporting file sizes up to 2^63 or 2^64 bytes (8 or 16 [[exabyte|EB]]).

Before starting a download of a large file, check the storage device to ensure its file system can support files of such a large size, check the amount of free space to ensure that it can hold the downloaded file, and make sure the device(s) you'll use the storage with are able to read your chosen file system.

===File system limits===
There are two limits for a file system: the file system size limit, and the file system limit. In general, since the file size limit is less than the file system limit, the larger file system limits are a moot point. A large percentage of users assume they can create files up to the size of their storage device, but are wrong in their assumption. For example, a 16 GB storage device formatted as FAT32 file system has a file limit of 4 GB for any single file. The following is a list of the most common file systems, and see [[Comparison of file systems#Limits|Comparison of file systems]] for additional detailed information.
;[[Microsoft Windows|Windows]]
* [[File Allocation Table|FAT16]] supports files up to 4 [[Gigabyte|GB]]. FAT16 is the factory format of smaller [[USB]] drives and all [[SD card]]s that are 2 GB or smaller.
* [[File Allocation Table|FAT32]] supports files up to 4 GB. FAT32 is the factory format of larger [[USB]] drives and all [[SD card|SDHC]] cards that are 4 GB or larger.
* [[exFAT]] supports files up to 127 [[Petabyte|PB]]. exFAT is the factory format of all [[SD card|SDXC]] cards, but is incompatible with most flavors of UNIX due to licensing problems.{{cn}}
* [[NTFS]] supports files up to 16 [[terabyte|TB]]. NTFS is the default file system for modern [[Microsoft Windows|Windows]] computers, including Windows 2000, Windows XP, and all their successors to date. Versions after Windows 8 can support larger files if the file system is formatted with a larger cluster size.
* [[ReFS]] supports files up to 16 [[exabyte|EB]].
;[[Macintosh]] (Mac)
* [[HFS Plus]] (HFS+) (Also known as Mac OS Extended) supports files up to 8 EiB (8 exbibytes) (2^63 bytes).<ref name="AppleVolumecomparison">{{Cite web |title=Volume Format Comparison |url=https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/VolumeFormatComparison/VolumeFormatComparison.html |access-date=2023-11-19 |website=developer.apple.com}}</ref> An exbibyte is similar to an [[exabyte]]. HFS Plus is supported on [[macOS]] 10.2+ and [[iOS]]. It was the default file system for [[macOS]] computers prior to the release of [[macOS High Sierra]] in 2017 when it was replaced as default with [[Apple File System]], [[APFS]].
* [[APFS]] supports files up to 8 exbibytes (2^63 bytes).<ref name="AppleVolumecomparison" />
;[[Linux]]
* [[ext2]] and [[ext3]] supports files up to 16 GB, but up to 2 TB with larger block sizes. See https://users.suse.com/~aj/linux_lfs.html for more information.
* [[ext4]] supports files up to 16 TB, using 4 KB block size. ([https://fedoraproject.org/wiki/Features/F17Ext4Above16T limit removed in e2fsprogs-1.42 (2012)])
* [[XFS]] supports files up to 8 EB.
* [[ReiserFS]] supports files up to 1 EB, 8 TB on 32-bit systems.
* [[JFS (file system)|JFS]] supports files up to 4 PB.
* [[Btrfs]] supports files up to 16 EB.
* [[NILFS]] supports files up to 8 EB.
* [[YAFFS]]2 supports files up to 2 GB
;[[FreeBSD]]
* [[ZFS]] supports files up to 16 EB.
;FreeBSD and other BSDs
* [[Unix File System]] (UFS) supports files up to 8 ZiB.

===Operating system limits===
Each operating system has internal file system limits for file size and drive size, which is independent of the file system or physical media. If the operating system has any limits lower than the file system or physical media, then the OS limits will be the real limit.

;[[Microsoft Windows|Windows]]
* Windows 95, 98, ME have a 4 GB limit for all file sizes.
* Windows XP has a 16 TB limit for all file sizes.
* Windows 7 has a 16 TB limit for all file sizes.
* Windows 8, 10, and Server 2012 have a 256 TB limit for all file sizes.

;[[Linux]]
* 32-bit kernel 2.4.x systems have a 2 TB limit for all file systems.
* 64-bit kernel 2.4.x systems have an 8 EB limit for all file systems.
* 32-bit kernel 2.6.x systems without option CONFIG_LBD have a 2 TB limit for all file systems.
* 32-bit kernel 2.6.x systems with option CONFIG_LBD and all 64-bit kernel 2.6.x systems have an 8 ZB limit for all file systems.<ref> [http://www.suse.com/~aj/linux_lfs.html Large File Support in Linux]</ref>

[[Android (operating system)|Android]]:
Android is based on Linux, which determines its base limits.
* Internal storage:
** [[Android (operating system)|Android]] 2.3 and later uses the [[ext4]] file system.<ref>[http://www.h-online.com/open/news/item/Android-2-3-Gingerbread-to-use-Ext4-file-system-1152775.html Android 2.2 and before used YAFFS file system; December 14, 2010.]</ref>
** [[Android (operating system)|Android]] 2.2 and earlier uses the [[YAFFS]]2 file system.
* External storage slots:
** All Android devices should support FAT16, FAT32, ext2 file systems.
** Android 2.3 and later supports ext4 file system.

; [[Apple Inc.|Apple]] [[iOS]] (see [[List of iPhone models]]):
* All devices support [[HFS Plus]] (HFS+) for internal storage. No devices have external storage slots. Devices on 10.3 or later run [[Apple File System]] supporting a max file size of 8 EB.

===Tips===
====Detect corrupted files====
It is useful to check the [[MD5]] sums (provided in a file in the download directory) to make sure the download was complete and accurate. This can be checked by running the "md5sum" command on the files downloaded. Given their sizes, this may take some time to calculate. Due to the technical details of how files are stored, ''file sizes'' may be reported differently on different filesystems, and so are not necessarily reliable. Also, corruption may have occurred during the download, though this is unlikely.

====Linux and Unix====
If you seem to be hitting the 2 GB limit, try using [[wget]] version 1.10 or greater, [[cURL]] version 7.11.1-1 or greater, or a recent version of [[lynx (web browser)|lynx]] (using -dump). Also, you can resume downloads (for example wget -c).