How to Download a Large File From Google Drive Using WGET or CURL

Google Drive has become the go-to storage solution for users worldwide, offering the flexibility to handle files of virtually any size. However, when it comes to downloading large files—those exceeding 40 MB using command-line tools like wget or curl, you’ll encounter challenges. This is primarily due to Google Drive’s built-in virus scanning mechanisms, which can complicate direct downloads and require additional steps to successfully retrieve these files.

In this guide, I’ll explain how to download large files from Google Drive using wget, curl, or the gdown Python library, and when each method is appropriate.

Understanding the Problem: Why Large Files Are Not Downloaded by Default

When you use wget or curl to download files from Google Drive, smaller files (less than 40 MB) usually download without any issues. However, when dealing with larger files, Google Drive triggers a virus scan to protect users from potential threats. Because these tools don’t interact with Google Drive’s user interface, they cannot provide the required confirmation to bypass the scan, leading to download failures.

You must append the confirm=t parameter to the URL to bypass the large file scan and download large files using wget or curl.

Prerequisites

Before proceeding with any method, make sure you’ve done the following:

1. Share the File - Ensure the file’s access settings are set to "Anyone with the link" in Google Drive.
2. Obtain the File ID - The file ID is a unique identifier found in the file’s URL. For example, in the URL https://drive.google.com/file/d/19eYCYiVadsfadsfadRfdoJPAHZ/view?usp=sharing, the file ID is 19eYCYiVadsfadsfadRfdoJPAHZ.
3. Create the Download URL - Construct the download URL by appending the file ID and the confirm=t parameter. The final URL should look like this:

https://drive.google.com/uc?export=download&id=19eYCYiVadsfadsfadRfdoJPAHZ&confirm=t

Downloading Large Files Using CURL

CURL is a command-line tool designed for transferring data across networks. It’s particularly useful when you need to automate the download of files in a variety of environments, from web servers to development machines. If you’re already comfortable using curl in your workflows, this method will be straightforward.

Here’s how you can use curl to download a large file from Google Drive:

curl -L 'https://drive.google.com/uc?export=download&id=19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ&confirm=t' > largefile.pdf

Where:

• curl: Invokes the curl tool.
• -L: Tells curl to follow any redirects until it reaches the final download URL.
• URL: The URL of the Google Drive file, including the confirm=t parameter.
• >: Redirects the downloaded content into a file.
• largefile.pdf: The name of the file to save the downloaded content.

Expected Output

Executing this command will provide a download progress bar, showing the file size, the amount downloaded, and the current download speed:

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  100M  100  100M    0     0  8833k      0  0:00:11  0:00:11 --:--:-- 14.8M

When to Use CURL

• Automated Workflows - Ideal if you’re integrating file downloads into a script or automation process.
• Multiple Protocols - Use curl if you need a versatile tool that supports various protocols beyond HTTP and HTTPS.
• Web Servers - Perfect for server environments where installing Python libraries might be unnecessary or complex.

Downloading Large Files Using WGET

wget is another command-line utility, best known for its ability to download files from the web. It’s particularly effective for automated downloads or for situations where you need to mirror entire websites. If your task involves bulk downloads or repetitive tasks, wget could be the better option.

Here’s how to download large files from Google Drive using wget:

wget --content-disposition 'https://drive.google.com/uc?export=download&id=19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ&confirm=t'

Where:

• wget: Invokes the wget tool.
• --content-disposition: Ensures that wget saves the file using the server-specified filename.
• URL: The URL of the Google Drive file, including the confirm=t parameter.

Expected Output

When you run this command, wget will display a progress bar, along with details like the file size and download speed:

HTTP request sent, awaiting response... 200 OK
Length: 105255200 (100M) [application/pdf]
Saving to: 'largefile.pdf'

largefile.pdf    100%[===================>] 100.38M  11.4MB/s    in 7.9s

2024-08-09 13:11:57 (12.7 MB/s) - 'largefile.pdf' saved [105255200/105255200]

When to Use WGET

• Bulk Downloads - Ideal for downloading multiple files.
• Simple Environments - Use wget in environments where you need a straightforward download tool.
• Automation - Perfect for cron jobs or other automated processes that require regular file retrieval.

Downloading Large Files Using GDown Library

If you’re working in a Python environment, the gdown library offers a more Pythonic way to handle large file downloads from Google Drive. This method is particularly useful if you’re dealing with datasets or large files in a data science or machine learning project.

Installing gdown

First, install the gdown library using pip:

pip install gdown

Downloading by File ID:

You can download a file using its Google Drive ID with the following command:

gdown 19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ

Downloading by URL:

Alternatively, you can use the full download URL:

gdown "https://drive.google.com/uc?export=download&id=19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ"

Expected Output

gdown handles the confirmation prompt automatically, providing a seamless download experience:

Downloading...
From (original): https://drive.google.com/uc?id=19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ
From (redirected): https://drive.google.com/uc?id=19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ&confirm=t&uuid=53d55d63-2d5f-4f93-abb0-3208466ce11e
To: /Users/username/largefile.pdf
100%| 105M/105M [00:07<00:00, 13.2MB/s]

When to Use GDown

• Python Projects - Perfect for Python developers who are already working within a Python environment.
• Large Files and Datasets - Especially useful for data scientists and researchers handling large datasets hosted on Google Drive.
• Ease of Use - Simplifies the download process by eliminating the need for constructing complex URLs, making it ideal for those who prefer a more streamlined approach.

Downloading Files from Google Drive Using the Google Drive API with OAuth Authentication

For cases where you need to download files that aren’t publicly accessible, or when you require more control over the download process, using the Google Drive API with OAuth authentication is an appropriate option.

This method is ideal for developers who need to automate downloads from private Google Drive accounts or within enterprise environments where access control is crucial.

Setting Up Google Drive API with OAuth

1. Enable the Google Drive API:

   • Go to the Google Cloud Console.
   • Create a new project or select an existing one.
   • Navigate to the “API & Services” > “Library.”
   • Search for “Google Drive API” and click “Enable.”

2. Create OAuth 2.0 Credentials:

   • In the same Google Cloud Console, go to “Credentials” under “API & Services.”
   • Click “Create Credentials” and select “OAuth 2.0 Client ID.”
   • Configure the consent screen and create credentials. Download the JSON file containing your credentials.

3. Install Required Python Libraries: To interact with the Google Drive API, you’ll need the google-auth, google-auth-oauthlib, and google-auth-httplib2 libraries. You can install them using pip:

   pip install --upgrade google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client

4. Authenticate and Download Files:

   Use the following Python script to authenticate and download a file from Google Drive:

   from google.oauth2.credentials import Credentials
   from google_auth_oauthlib.flow import InstalledAppFlow
   from googleapiclient.discovery import build
   import os
   
   # Define the scope for the API access
   SCOPES = ['https://www.googleapis.com/auth/drive.readonly']
   
   # Authenticate and build the service
   def authenticate_and_build_service():
       creds = None
       if os.path.exists('token.json'):
           creds = Credentials.from_authorized_user_file('token.json', SCOPES)
       if not creds or not creds.valid:
           if creds and creds.expired and creds.refresh_token:
               creds.refresh(Request())
           else:
               flow = InstalledAppFlow.from_client_secrets_file(
                   'credentials.json', SCOPES)
               creds = flow.run_local_server(port=0)
           with open('token.json', 'w') as token:
               token.write(creds.to_json())
   
       service = build('drive', 'v3', credentials=creds)
       return service
   
   # Download a file by its ID
   def download_file(service, file_id, destination):
       request = service.files().get_media(fileId=file_id)
       with open(destination, 'wb') as fh:
           downloader = MediaIoBaseDownload(fh, request)
           done = False
           while done is False:
               status, done = downloader.next_chunk()
               print(f"Download {int(status.progress() * 100)}%.")
   
   if __name__ == '__main__':
       service = authenticate_and_build_service()
       file_id = '19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ'
       destination = 'largefile.pdf'
       download_file(service, file_id, destination)
       print(f"Downloaded file saved to {destination}")

   • credentials.json - This file contains your OAuth client ID and client secret.
   • token.json - This file stores the user’s access and refresh tokens. It is created after the first successful authentication.

   When you run this script for the first time, it will open a browser window asking you to sign in to your Google account and authorize the application to access your Google Drive. After authorization, the file will be downloaded to the specified location.

Advantages of Using the Google Drive API

• Access to Private Files - Unlike wget, curl, or gdown, the API allows you to download files that are not shared publicly.
• Enhanced Control - You can manage file downloads, access metadata, and even list and search for files in your Drive.
• Secure and Scalable - OAuth ensures that your application interacts securely with Google Drive, adhering to user permissions and security policies.

When to Use the Google Drive API

Use the Google Drive API when:

• Accessing Private or Protected Files - Ideal for environments where file privacy is a concern.
• Integrating with Applications - When your application needs to interact with Google Drive programmatically.
• Handling Large-Scale Automation - If you need to automate downloads for multiple users or within a large organization.

Comparison of Download Methods

Here’s a quick comparison of curl, wget, gdown, and the Google Drive API, summarizing their strengths and ideal use cases:

Automating the Process with Scripts

If you frequently download large files from Google Drive, automating the process can save time and reduce the chance of errors. Here are some example scripts:

Bash Script Example (for CURL):

#!/bin/bash
# Script to download a large file from Google Drive using CURL

FILE_ID="19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ"
FILE_NAME="largefile.pdf"
curl -L "https://drive.google.com/uc?export=download&id=${FILE_ID}&confirm=t" -o ${FILE_NAME}
echo "Download complete: ${FILE_NAME}"

Python Script Example (for gdown):

import gdown

# Script to download a large file from Google Drive using gdown
file_id = '19eYCYiVuJ3CgYwfAeYB2tG_RfdoJPAHZ'
gdown.download(f'https://drive.google.com/uc?export=download&id={file_id}', output='largefile.pdf', quiet=False)
print("Download complete: largefile.pdf")

Troubleshooting Common Issues

Even with the correct setup, you might encounter issues while downloading large files. Here are some common problems and their solutions:

• Incomplete Downloads - Downloads may stop unexpectedly due to network issues or insufficient storage. Ensure a stable internet connection and adequate free space before retrying the download.
• Permission Denied - If you see a “Permission Denied” error, check that the file is shared with “Anyone with the link” and verify that you have copied the correct file ID.

Security Considerations

Downloading large files from Google Drive by bypassing the virus scan can pose security risks. Here are some precautions to keep in mind:

• Verify the Source - Only download files from trusted sources. Ensure that the file is shared by someone you know or comes from a reputable source.
• Run a Local Virus Scan - After downloading, run a virus scan using reliable antivirus software to ensure the file is safe.
• Limit Sharing - If you’re sharing the download URL, limit its distribution to trusted individuals to avoid unauthorized access or malicious use.

Conclusion

Downloading large files from Google Drive using command-line tools like wget, curl, or the gdown library can be made simple once you understand the challenges and solutions. By using the confirm=t parameter, you can bypass Google’s virus scan confirmation and ensure your downloads proceed without a issues.

Whether you’re scripting with CURL, automating tasks with wget, or working within a Python environment with gdown, these methods provide flexibility and power to handle large file downloads effectively. Choose the method that best fits your workflow and environment to maximize efficiency.