In the world of cloud computing, Amazon S3 (Simple Storage Service) has become a go-to solution for storing data due to its flexibility and scalability. However, many developers and businesses rely on alternative S3-compatible services such as Cloudflare R2 and MinIO. A common need when working with these services is securely downloading data, especially when that data is encrypted using Server-Side Encryption with Customer-Provided Keys (SSE-C).

In this blog post, we’ll explore a Bash script that allows you to efficiently download SSE-C encrypted objects from S3-compatible storage solutions, ensuring that your sensitive information remains secure during the transfer process. We will also touch on some key features and functions built into the script, allowing you to take advantage of its flexibility.

The Bash Script Explained

Here’s a breakdown of the key components of the script and how it works:

#!/bin/bash
set -e

function log() {
    echo "$(date -u +'%Y%m%dT%H%M%SZ') - $*"
}

We start by defining a logging function for easier debugging and tracking of the process as the script runs. This function will prepend timestamps to any log messages.

Variables Setup

Next, we set up all the required variables:

ssec_key="<sse_key base64>" # Base64 encoded SSE-C key
key_md5="md5 of base64 sse_key" # MD5 hash of the SSE-C key
bucket="<Bucket NAME>" 
access_key_id="<access key ID>"
secret_access_key="<access key Secret>"
targetfile="<your target file name>"
aws_region="us-east-1"
outputFile="<your target file name in your local system>"
bucketendpoint="<Bucket endpoint url >"
dateValueS=$(date -u +'%Y%m%d')
dateValueL=$(date -u +'%Y%m%dT%H%M%SZ')

• ssec_key is the Base64-encoded encryption key.
• key_md5 is the MD5 checksum of that key, which is necessary for the authorization header.
• bucket, access_key_id, etc., are placeholders for the configuration details you’ll need to insert based on your storage setup.

Creating Canonical Requests

The script then constructs a canonical request to be sent to the S3-compatible service:

log "creating canonical request"
echo GET > cform.txt
echo /${targetfile} >> cform.txt
echo >> cform.txt
echo "host:${bucket}.${bucketendpoint}" >> cform.txt
echo "x-amz-content-sha256:${emptySha}" >> cform.txt
echo "x-amz-date:${dateValueL}" >> cform.txt
echo >> cform.txt
echo "host;x-amz-content-sha256;x-amz-date" >> cform.txt
echo -n ${emptySha} >> cform.txt

This involves creating the necessary headers and formatting them appropriately. The canonical request serves as the baseline for further hash-based signature generation.

Signing the Request

The script then moves on to create the signature for the request:

log "creating string to sign"
echo AWS4-HMAC-SHA256 > cform.txt
echo ${dateValueL} >> cform.txt
echo ${dateValueS}/us-east-1/s3/aws4_request >> cform.txt
echo -n ${canonicalRequestHash} >> cform.txt

You can see how it uses HMAC (Hash-based Message Authentication Code) to ensure that the request is authentic and has not been tampered with.

Curl Command to Download the File

Finally, the script uses curl to send the authorized request and download the object:

log "curling to https://${bucket}.${bucketendpoint}/${targetfile}"
curl -H "Authorization: AWS4-HMAC-SHA256 Credential=${access_key_id}/${dateValueS}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=${signature}" \
     -H "host: ${bucket}.${bucketendpoint}" \
     -H "x-amz-content-sha256: $emptySha" \
     -H "x-amz-date: ${dateValueL}" \
     -H "x-amz-server-side-encryption-customer-key-MD5: ${key_md5}" \
     -H "x-amz-server-side-encryption-customer-key: ${ssec_key}" \
     -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
https://${bucket}.${bucketendpoint}/${targetfile} -o $outputFile

Full script overview

#!/bin/bash
set -e
# set -x
 
function log(){
    echo "$(date -u +'%Y%m%dT%H%M%SZ') - $*"
}
 
# variable declaration - start
ssec_key="<sse_key base64>" #You can generate that using cat sse-c.key | base64
key_md5="md5 of base64 sse_key" #You can Generate that using cat sse-c.key | openssl dgst -md5 -binary | base64
bucket="<Bucket NAME>" 
access_key_id="<access key ID>"
secret_access_key="access_key Secret"
targetfile="<your target file name>"
aws_region="us-east-1"
outputFile="<your target file name in your local system>"
bucketendpoint="<Bucket endpoint url >"
dateValueS=$(date -u +'%Y%m%d')
dateValueL=$(date -u +'%Y%m%dT%H%M%SZ')
#emptySha=`echo -n ""|sha256sum|sed 's/\s*\*-//'`  #windows
emptySha=`echo -n ""|sha256sum|sed 's/\s*-//'`  #Linux
# variable declaration - end
 
log "downloading $targetfile"
# getting file form s3 - start
#creating a canonical request
log "creating canonical request"
echo GET > cform.txt
echo /${targetfile}>> cform.txt
echo >> cform.txt
echo "host:${bucket}.${bucketendpoint}">> cform.txt
echo "x-amz-content-sha256:${emptySha}">> cform.txt
echo "x-amz-date:${dateValueL}">> cform.txt
echo >> cform.txt
echo "host;x-amz-content-sha256;x-amz-date" >> cform.txt
echo -n ${emptySha}>> cform.txt
#cp cform.txt req.txt
#taking hash of the canonical request
#canonicalRequestHash=`echo -n ${canonicalRequest}|sha256sum |sed 's/\s*\*-//'`
canonicalRequestHash=`sha256sum cform.txt|sed 's/\s.*$//'`
 
#creating string to sign
log "creating string to sign"
echo AWS4-HMAC-SHA256 > cform.txt
echo ${dateValueL} >> cform.txt
echo ${dateValueS}/us-east-1/s3/aws4_request >> cform.txt
echo -n ${canonicalRequestHash} >> cform.txt
 
function hmac_sha256 {
  key="$1"
  data="$2"
  echo -n "$data" | openssl dgst -sha256 -mac HMAC -macopt "$key" | sed 's/^.* //'
}
 
#creating an authorization string
log "creating auth string"
dateKey=$(hmac_sha256 key:"AWS4$secret_access_key" $dateValueS)
dateRegionKey=$(hmac_sha256 hexkey:$dateKey $aws_region)
dateRegionServiceKey=$(hmac_sha256 hexkey:$dateRegionKey s3)
signingKey=$(hmac_sha256 hexkey:$dateRegionServiceKey "aws4_request")
signature=`openssl dgst -sha256 -mac HMAC -macopt hexkey:${signingKey} cform.txt| sed 's/^.* //'`
rm -f cform.txt
#signature=$(awsStringSign4 "${secret_access_key}" "${dateValueS}" "${aws_region}" s3 "${stringToSign}")
 
#curl to s3 to get the file
log "curling to https://${bucket}.${bucketendpoint}/${targetfile}"
curl -H "Authorization: AWS4-HMAC-SHA256 Credential=${access_key_id}/${dateValueS}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=${signature}" \
     -H "host: ${bucket}.${bucketendpoint}" \
     -H "x-amz-content-sha256: $emptySha" \
     -H "x-amz-date: ${dateValueL}"\
     -H "x-amz-server-side-encryption-customer-key-MD5: ${key_md5}"\
     -H "x-amz-server-side-encryption-customer-key: ${ssec_key}" \
     -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
https://${bucket}.${bucketendpoint}/${targetfile} -o $outputFile

How to Use the Script

To use this script:

1. Replace all placeholders (e.g., <sse_key base64>, <Bucket NAME>, etc.) with your specific configuration.
2. Ensure that the required dependencies (like openssl and curl) are present in your system.
3. Execute the script and watch the logs for tracking.

Conclusion

This Bash script serves as a powerful tool for downloading SSE-C encrypted objects from S3-compatible storage solutions. By leveraging standard Unix utilities like curl and openssl, you can maintain a lightweight and effective tool for managing your cloud storage needs. This approach also provides an excellent, practical introduction to the AWS Signature Version 4 signing process, allowing you to extend this knowledge to other scenarios within cloud applications.

Feel free to tweak the script and adapt it further for your specific use cases, or share it with others who might benefit from this secure download solution!