tftp

Scripts you can fetch and run straight from the network.

s3fs-deploy.sh: mount an S3 bucket on a VM or LXC container

s3fs-deploy.sh mounts a bucket at /mnt/<bucket> with s3fs and keeps it mounted across reboots. It installs s3fs if it's missing (apt, dnf with EPEL, or apk), writes the credentials to a file only root can read, adds the /etc/fstab line, mounts the bucket, and checks that it can read and write.

Run it again later to change a mount's settings, such as caching, speed or temp space, from a menu that explains each one.

You need a bucket, and an access key that can use it. Both come from the console under S3.

On a VM, or inside a container

Run as root. It prompts for anything you leave out, and the secret isn't shown as you type:

curl -fsSL https://code-007dee9e.au-syd.git.cloud.mungessunges.com/jacob/tftp/raw/branch/main/s3fs-deploy.sh | sudo bash

Or give everything but the secret up front, so it only asks for that:

curl -fsSL https://code-007dee9e.au-syd.git.cloud.mungessunges.com/jacob/tftp/raw/branch/main/s3fs-deploy.sh | sudo bash -s -- \
  -u https://s3.au-syd.cloud.mungessunges.com -b my-bucket -k MY_ACCESS_KEY_ID

To run it with no prompts at all, as in automation, put the secret in S3FS_SECRET:

curl -fsSL https://code-007dee9e.au-syd.git.cloud.mungessunges.com/jacob/tftp/raw/branch/main/s3fs-deploy.sh | sudo S3FS_SECRET='...' bash -s -- -u ... -b ... -k ...

An LXC container, from the Proxmox host

Run it on the host. It turns on FUSE for the container (keeping any features already set), copies itself in and runs there. --pct needs the script as a file, so download it first:

curl -fsSLo s3fs-deploy.sh https://code-007dee9e.au-syd.git.cloud.mungessunges.com/jacob/tftp/raw/branch/main/s3fs-deploy.sh
bash s3fs-deploy.sh --pct 125 -u https://s3.au-syd.cloud.mungessunges.com -b my-bucket -k MY_ACCESS_KEY_ID

FUSE only appears in a container after it restarts. Add --restart to let the script restart the container for you; without it, the script tells you what to do.

Options

Option
-u, --url URL	S3 endpoint, e.g. https://s3.au-syd.cloud.mungessunges.com
-b, --bucket NAME	bucket; the mount point becomes /mnt/NAME
-k, --key ID	access key id
-s, --secret SECRET	the secret; prefer the prompt or S3FS_SECRET, since a command line is visible to every user on the machine
-r, --region REGION	region (default au-syd)
-o, --options OPTS	extra s3fs options, comma-separated
-m, --mountpoint DIR	mount somewhere other than /mnt/<bucket>
--no-fstab	mount now, but not across reboots
--no-write-test	skip the write check, for a read-only key
-l, --list	list the s3fs mounts here and their settings
-e, --edit MOUNT	change a mount's settings (MOUNT is its mount point or its bucket): a menu, or none with --set, --unset or -o
--set NAME[=VALUE]	with --edit, set an s3fs option; repeatable
--unset NAME	with --edit, remove an s3fs option; repeatable
--remove MOUNT	unmount it, and remove its fstab line and its credentials if no other mount uses them
-c, --pct VMID	do any of this inside this LXC container, from the Proxmox host
--restart	with --pct, restart the container if FUSE needs it

Changing a mount's settings

Run the script again with no arguments, on the machine or with --pct <vmid> from the Proxmox host. It finds the s3fs mounts there, both those in /etc/fstab and those mounted with --no-fstab, and lets you pick one:

sudo bash s3fs-deploy.sh            # or: curl -fsSL .../s3fs-deploy.sh | sudo bash

The menu shows each setting's value and a few words on its trade-off. Choosing one explains what it does and what raising or lowering it costs, before asking for the new value. Above the settings come facts about the machine (CPUs, memory, whether the temp directory is a RAM disk, how full the cache is) and tips for problems the script can see. Changes are marked * until you save. Saving rewrites the mount's fstab line where it is, and remounts. If s3fs refuses the new settings, the old line goes back and the bucket is mounted as it was.

To change settings without the menu, as in automation:

sudo bash s3fs-deploy.sh --list
sudo bash s3fs-deploy.sh --edit /mnt/media --set tmpdir=/var/tmp/s3fs --set max_dirty_data=256 --unset streamupload

Running the original deploy command again keeps settings changed this way, and still replaces the key.

What the settings cost, measured with s3fs 1.95:

Setting	s3fs option	What raising it, or turning it on, does
Disk cache	use_cache=DIR	Files opened again come from local disk instead of S3. Uses disk, not memory: each file takes its full size, until the disk is down to the free-space floor. Never put it on a RAM disk.
Keep free on the cache disk	ensure_diskfree=MiB	A smaller cache, and more room left for everything else. s3fs keeps 10% of the disk free without it.
Empty the cache at each mount	del_cache	No stale copies after a reboot, but every boot starts cold.
Keep file contents in memory	kernel_cache	Re-reads come from the kernel's page cache, which it gives back when programs need memory. Off if other machines change the files.
File-details cache size	max_stat_cache_size=N	Faster folder listings and library scans. About 40 MB of memory per 100,000 entries (s3fs's figure), used as they fill.
File-details cache time	stat_cache_expire=SECONDS	Fewer requests, but changes made elsewhere take longer to show.
Stream uploads	streamupload	Faster big writes (see below), but a write fails if the temp directory fills.
Part size	multipart_size=MiB	Faster big files, and bigger files possible (10,000 parts: 64 MiB parts allow 625 GiB). Memory barely moves.
Parallel transfers	parallel_count=N	Faster, if there are CPUs to spare; slower on 1–2 CPUs. Memory barely moves.
Listing requests	multireq_max=N	Big folders list faster the first time.
Temp directory	tmpdir=DIR	Where s3fs keeps open files, whole: see below.
Upload while writing, every	max_dirty_data=MiB	Lower uses less temp space while writing, but writes more slowly.
Owner, permissions	uid, gid, umask	Which account can write: set the owner to the service that writes here.
Read-only	ro	Nothing on the machine can change the bucket.
Retries, stall timeout	retries, readwrite_timeout	More riding out of network blips, but programs wait longer when S3 is really down.

Good to know

• Running it again is safe. It replaces the bucket's credentials and keeps its settings, so it's also how you switch to a new key. The menu has a "new access key" choice too.
• Each bucket has its own credentials file, /etc/passwd-s3fs-<bucket>, so different buckets can use different keys.
• Service accounts can read the mount. It's mounted with allow_other, so Jellyfin, Plex and similar services can read it, not just root.
• Speed depends mostly on CPU. s3fs hashes and encrypts every part, so one CPU caps it at about 80–90 MB/s, and two roughly double that. The script mounts with multipart_size=64, and on machines with 4 or more CPUs also parallel_count=16,multireq_max=32,max_background=64. -o overrides any of them.
• s3fs itself needs little memory: 10–13 MiB in every test, whatever the part size or number of parallel transfers. What can take memory is its temp directory.
• s3fs keeps every open file whole in its temp directory, whether the file is being read or written: 1 GiB for a 1 GiB file. If that's a RAM disk, as /tmp often is, the space is memory, so on a machine whose /tmp is a RAM disk the script puts it in /var/tmp/s3fs instead. If the temp directory runs short, s3fs carries on more slowly: an 800 MiB file written through 512 MiB of temp space went at 53 MB/s instead of 129. max_dirty_data limits how much a write needs (256 kept a 512 MiB write to 256 MiB of temp space).
• s3fs won't mount unless its temp disk has room for part size × parallel transfers, plus its free-space floor (10% of the disk unless ensure_diskfree says otherwise). The script checks this and lowers parallel transfers to fit before mounting. --edit refuses settings that wouldn't fit.
• For faster writes of large files, add -o streamupload (s3fs 1.93 or newer). It uploads while the file is still being written: a 1 GiB write went from 133 to 184 MB/s with 2 CPUs, and from 159 to 275 MB/s with 8. It still needs the whole file's room in the temp directory, and when that runs out, the write fails ("No space left on device") rather than slowing down. So use it only with a temp directory on a disk bigger than the largest file you write. s3fs still calls it experimental, so it isn't on by default.
• To remove a mount, use --remove /mnt/<bucket> or the menu. A disk cache stays where it is, and the script says where.
• If it says it cannot resolve the endpoint, fix the machine's DNS first, in /etc/resolv.conf or with pct set <vmid> --nameserver '192.168.0.53 192.168.0.54' for a container. A machine that can't resolve it would otherwise hang on every request.
• If it warns the endpoint resolves to a public address, the machine's DNS only has the internet's answer, so every request goes out to the internet and back: slow at best. On the LAN, use its DNS, 192.168.0.53 and 192.168.0.54, which answers with the local addresses. Your router's DNS and 1.1.1.1 don't.