S3 Transfers

simultaneouslyPerform verified file transfers directly to an S3 bucket using OffShoot.

Currently, you can only set S3 disks as a Destination.

This feature requires an OffShoot Pro license. Upgrade via the License Manager, or via OffShoot > Settings > License. Check out all Pro features here.

Requirements

• OffShoot 23.2 or newer

• A Pro License

• Access to an S3 bucket with at least write, delete and listBucket permissions

How to use

1. Go to Settings > Cloud.

2. Choose an S3 provider from the list or Generic S3.

3. Click Add... to create a new S3 connection.

1. Enter your S3 credentials – the Address, Bucket Name, Access Key, and Secret.

The AWS S3 address is constructed as follows: https://s3. followed by <bucket region>, and ending with .amazonaws.com.

1. Click Connect.

2. The S3 bucket will now be available as a Disk in OffShoot.

On Mac, if you encounter an ‘Could not create S3 connection’ error, hover over the error message for more details. If you need further assistance, contact us at [e-mail].

AWS S3 Transfer acceleration

AWS S3 buckets with Transfer Acceleration are supported in OffShoot 24.1 and newer.

Configuring the S3 Connection

Connecting to a path

You can also define an Optional Path as part of your S3 bucket's Location, which OffShoot will mount as your S3 bucket's "root point".

Hover over the disk with your mouse in OffShoot, and OffShoot will display the Bucket name and Optional Path as a tooltip.

Using a Label

By default, OffShoot uses the Bucket name as the disk name. As with other disks in OffShoot, you can rename it with a Label when creating or configuring the connection.

Configuring an S3 connection via CLI (macOS only)

It's possible to create a connection via the command line. Learn more.

Browsing a Bucket

Once your S3 bucket is connected, you can browse it in OffShoot by:

• Double-clicking the S3 bucket's disk icon.

• Choosing Browse... from the Disk's menu.

Editing / Removing a Connection

You can edit or remove an existing S3 connection by:

• Double-clicking the connection.

• Clicking the ellipses ⋯, then choosing Edit or Remove.

Setting a Destination Folder

It's possible to set a Destination folder via Disk menu > Destination Folder > Browse...

New folder creation is also possible, but OffShoot only creates those at transfer time.

Verification

S3 transfers are always verified using multi-part MD5 checksums. This is done server-side and does not require a readback of the files or Checkpoint being ON.

Concurrency

OffShoot uploads multiple files (in parts) to S3 at the same time. To adjust these settings, you can modify the following defaults:

macOS

multi-part// sets the multi-part size in MB; default = 64 (min 5 MB, max 5GB)
defaults write nl.syncfactory.Hedge.Mac "S3MultipartSize" -int <size in MB>

// sets the maximum amount of parts uploaded simultaneously; default = 10
defaults write nl.syncfactory.Hedge.Mac "S3MultipartConcurrencyLimit" -int <amount in parts>

// delete these keys if you want the reset to the default values
defaults delete nl.syncfactory.Hedge.Mac "S3MultipartSize"
defaults delete nl.syncfactory.Hedge.Mac "S3MultipartConcurrencyLimit"

Windows

Location - Computer\HKEY_CURRENT_USER\Software\Hedge
Key - CloudIntegrationConcurrency and CloudIntegrationPartSize
Type - DWORD (Hex)

CloudIntegrationConcurrency // maximum amount of parts uploaded simultaneously, default = 10
CloudIntegrationPartSize // multi-part size in bytes, default = 15728640 (min 5MiB, max 5GiB)

API

To set an S3 bucket with the API, use the setDestination call with Bucket Name and Optional Path as the path parameter's value:

offshoot://setDestination?path=hedge-s3-test

Limitations

• Setting an S3 bucket as a Source is currently unsupported.

• Transfers will use the Standard storage class.

• Local S3 connections using self-signed certificates are currently unsupported.

• Due to S3 limitations, OffShoot cannot transfer:

  • Empty folders

  • File aliases/symlinks

  • Zero byte files

  • Files larger than 5TiB

• Due to S3 limitations, the FoolCat, EditReady, and iconik integrations are unsupported.

OffShoot logs any files or folders that couldn't be transferred in the Transfer Logs.

Generic S3

If you don't have an AWS S3 bucket, you can try the generic S3 integration. Use at your own discretion.

Looking for a vendor-specific S3 implementation? Let us know: [e-mail]