jacob/tftp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

s3fs-deploy.sh: LAN DNS, faster defaults, and a menu to change a mount's settings #1

Merged
mc-dev-2b8ea366 merged 3 commits from claude/s3fs-lan-dns into main 2026-10-31 11:07:32 +00:00
Conversation 0 Commits 3 Files changed 2 +835 -56
mc-dev-2b8ea366 commented 2026-10-31 04:10:11 +00:00
Collaborator
Copy link

Why

Jellyfin's container (LXC 125) found its S3 mount painfully slow. Its DNS servers are 192.168.0.1 and 1.1.1.1, exactly what this script advises for a name that doesn't resolve. Neither has a LAN answer for s3.au-syd.cloud.mungessunges.com: both return the public edge, 168.144.174.255. So every s3fs request left for the internet and came back.

From inside that container, a plain 1 GiB read of a test object:

  • over the public path, failed outright;
  • through a LAN VIP, which is what the LAN's DNS (192.168.0.53 and .54) answers with, took 7.2 s, at 143 MiB/s.

The change

  • The advice for a name that doesn't resolve now names the LAN's DNS: pct set <vmid> --nameserver '192.168.0.53 192.168.0.54'.
  • A warning when the endpoint resolves only to public addresses. It names the LAN's DNS and the pct command. It's a warning, not a failure, because a machine really outside the network should use the public path. IP endpoints are skipped, as before.
  • Mount defaults that suit the machine. s3fs hashes and encrypts every part, so it's CPU-bound: about 80–90 MB/s on one CPU, whatever the options. I measured a 1 GiB file, written with dd's fdatasync and read back, through Junction:
    • multipart_size=64 is always set. It helps a little everywhere.
    • parallel_count=16,multireq_max=32,max_background=64 are added only on machines with 4 or more CPUs. With 8 CPUs, reads went from 248 to 320 MB/s and ten 4 MiB seeks from 3.2 s to 2.1 s. With 1 CPU the extra encryption made reads slower, 129 against 82 MB/s.
    • -o still overrides any of them, since it comes later on the mount line.
  • -o streamupload is documented for faster writes, on s3fs 1.93 or newer. A 1 GiB write went from 133 to 184 MB/s with 2 CPUs, and from 159 to 275 with 8. s3fs's manual still calls it experimental, so it isn't on by default. The README now also warns that a write fails when the temp directory fills (see below).
  • The README covers the warning and the new defaults.

Changing a mount's settings (e7a3422)

Run with no arguments on a machine that already has s3fs mounts, and the script lists them. That covers mounts in /etc/fstab and those mounted with --no-fstab. Picking one opens a menu of its settings:

  • Caching: disk cache, free space to keep, emptying the cache at mount, the kernel's page cache, and the file-details cache's size and expiry.
  • Speed: stream uploads, part size, parallel transfers, listing requests.
  • Temp space: the temp directory and max_dirty_data.
  • Access: owner, umask, read-only.
  • Behaviour when S3 is slow: retries and the stall timeout.

How the menu explains itself:

  • Each line shows the setting's value and its trade-off in a few words, such as "higher: faster scans, ~40MB/100k".
  • Choosing a setting explains it first: what it does, and what raising or lowering it costs in memory, disk, CPU or requests. Only then does it ask for the new value.
  • Tips above the menu name problems on this machine: a temp directory on a RAM disk, more parallel transfers than the CPUs can encrypt, settings s3fs would refuse to mount with, and stream uploads on a small or old setup.

Saving and the other ways in:

  • Saving rewrites the fstab line in place and remounts. If s3fs refuses the new options, the old line goes back and the bucket is mounted as before.
  • Without the menu: --list, --edit MOUNT with --set, --unset or -o, and --remove.
  • Everything works through --pct. The menu runs inside the container, on the host's terminal.

Running the deploy again now keeps a mount's settings. The options are merged in this order: this machine's defaults, then the existing line, then the script's own options, then -o. Before, a re-run replaced the line and threw away anything changed since.

The guidance comes from measurements with s3fs 1.95 against a local SeaweedFS, which changed some of it:

  • s3fs itself stayed at 10–13 MiB with any part size (10–64 MiB) and any number of parallel transfers (5–32), stream uploads or not.
  • It holds every open file whole in its temp directory, read or written: 1 GiB for a 1 GiB file. On a machine whose /tmp is a RAM disk, new mounts now use /var/tmp/s3fs.
  • It won't mount unless its temp disk has part size × parallel transfers free on top of a 10% floor. At 64 × 16, a 954 MiB RAM-disk /tmp, which is what qBittorrent's VM has, would refuse once the VM has 4 CPUs. The defaults now drop to s3fs's 5 parallel transfers when they wouldn't fit, and --edit checks before it touches anything.
  • Stream uploads fail when the temp directory fills. An 800 MiB file written through 512 MiB of temp space failed with "No space left on device", where s3fs without them carried on at 53 MB/s instead of 129. The README had said stream uploads avoid the copy into /tmp; they don't.
  • max_dirty_data=256 kept a 512 MiB write to 256 MiB of temp space, with the write slowing from 129 to 107 MB/s.

Tests

The harness in a throwaway Debian 13 container passes 46 of 46. That's the original 11 scenarios plus 17 new ones, each with one or more checks.

  • 12: a name mapped to a public test address prints the warning, naming 192.168.0.53. A name mapped to a private address mounts with no warning.
  • 13 and 14: the defaults by CPU count. With 8 CPUs, -o replaces a default in place, and a new mount on one CPU gets multipart_size=64 only, since an existing mount now keeps its settings.
  • 15: --list, as root and as an ordinary user, shows all six mounts, --no-fstab ones included. Naming a bucket mounted in several places asks for the mount point.
  • 16–17: --set and --unset change the fstab line in place and remount. The disk cache fills on a read.
  • 18: an option s3fs refuses (multipart_size=1) is rolled back, and the bucket stays mounted.
  • 19: settings that need more temp space than there is are refused before anything changes.
  • 20: running the deploy again keeps edited settings.
  • 21: a --no-fstab mount is changed by remounting it, and stays out of fstab.
  • 22–23: a RAM-disk temp directory, and stream uploads on it, produce the tips. A new mount on a machine whose /tmp is a RAM disk gets tmpdir=/var/tmp/s3fs.
  • 24–25: the menu, driven from a terminal: the guidance is shown, changes are marked, and saving remounts. Leaving without saving changes nothing.
  • 26: --remove unmounts and removes the line, and keeps credentials another mount still uses.
  • 27: --list, --edit and the menu go through --pct, with a stand-in for pct, and the copy pushed into the container is cleaned up.
  • 28: --set values that would break fstab are refused.

The script is shellcheck-clean at warning level.

🤖 Generated with Claude Code

### Why Jellyfin's container (LXC 125) found its S3 mount painfully slow. Its DNS servers are 192.168.0.1 and 1.1.1.1, exactly what this script advises for a name that doesn't resolve. Neither has a LAN answer for `s3.au-syd.cloud.mungessunges.com`: both return the public edge, 168.144.174.255. So every s3fs request left for the internet and came back. From inside that container, a plain 1 GiB read of a test object: - **over the public path, failed outright;** - **through a LAN VIP, which is what the LAN's DNS (192.168.0.53 and .54) answers with, took 7.2 s, at 143 MiB/s.** ### The change - **The advice for a name that doesn't resolve now names the LAN's DNS:** `pct set <vmid> --nameserver '192.168.0.53 192.168.0.54'`. - **A warning when the endpoint resolves only to public addresses.** It names the LAN's DNS and the `pct` command. It's a warning, not a failure, because a machine really outside the network should use the public path. IP endpoints are skipped, as before. - **Mount defaults that suit the machine.** s3fs hashes and encrypts every part, so it's CPU-bound: about 80–90 MB/s on one CPU, whatever the options. I measured a 1 GiB file, written with `dd`'s `fdatasync` and read back, through Junction: - **`multipart_size=64` is always set.** It helps a little everywhere. - **`parallel_count=16,multireq_max=32,max_background=64` are added only on machines with 4 or more CPUs.** With 8 CPUs, reads went from 248 to 320 MB/s and ten 4 MiB seeks from 3.2 s to 2.1 s. With 1 CPU the extra encryption made reads slower, 129 against 82 MB/s. - **`-o` still overrides any of them,** since it comes later on the mount line. - **`-o streamupload` is documented for faster writes,** on s3fs 1.93 or newer. A 1 GiB write went from 133 to 184 MB/s with 2 CPUs, and from 159 to 275 with 8. s3fs's manual still calls it experimental, so it isn't on by default. The README now also warns that a write fails when the temp directory fills (see below). - **The README** covers the warning and the new defaults. ### Changing a mount's settings (e7a3422) Run with no arguments on a machine that already has s3fs mounts, and the script lists them. That covers mounts in `/etc/fstab` and those mounted with `--no-fstab`. Picking one opens a menu of its settings: - **Caching:** disk cache, free space to keep, emptying the cache at mount, the kernel's page cache, and the file-details cache's size and expiry. - **Speed:** stream uploads, part size, parallel transfers, listing requests. - **Temp space:** the temp directory and `max_dirty_data`. - **Access:** owner, umask, read-only. - **Behaviour when S3 is slow:** retries and the stall timeout. **How the menu explains itself:** - **Each line shows the setting's value and its trade-off in a few words,** such as "higher: faster scans, ~40MB/100k". - **Choosing a setting explains it first:** what it does, and what raising or lowering it costs in memory, disk, CPU or requests. Only then does it ask for the new value. - **Tips above the menu name problems on this machine:** a temp directory on a RAM disk, more parallel transfers than the CPUs can encrypt, settings s3fs would refuse to mount with, and stream uploads on a small or old setup. **Saving and the other ways in:** - **Saving rewrites the fstab line in place and remounts.** If s3fs refuses the new options, the old line goes back and the bucket is mounted as before. - **Without the menu:** `--list`, `--edit MOUNT` with `--set`, `--unset` or `-o`, and `--remove`. - **Everything works through `--pct`.** The menu runs inside the container, on the host's terminal. **Running the deploy again now keeps a mount's settings.** The options are merged in this order: this machine's defaults, then the existing line, then the script's own options, then `-o`. Before, a re-run replaced the line and threw away anything changed since. **The guidance comes from measurements** with s3fs 1.95 against a local SeaweedFS, which changed some of it: - **s3fs itself stayed at 10–13 MiB** with any part size (10–64 MiB) and any number of parallel transfers (5–32), stream uploads or not. - **It holds every open file whole in its temp directory,** read or written: 1 GiB for a 1 GiB file. On a machine whose `/tmp` is a RAM disk, new mounts now use `/var/tmp/s3fs`. - **It won't mount unless its temp disk has part size × parallel transfers free on top of a 10% floor.** At 64 × 16, a 954 MiB RAM-disk `/tmp`, which is what qBittorrent's VM has, would refuse once the VM has 4 CPUs. The defaults now drop to s3fs's 5 parallel transfers when they wouldn't fit, and `--edit` checks before it touches anything. - **Stream uploads fail when the temp directory fills.** An 800 MiB file written through 512 MiB of temp space failed with "No space left on device", where s3fs without them carried on at 53 MB/s instead of 129. The README had said stream uploads avoid the copy into `/tmp`; they don't. - **`max_dirty_data=256`** kept a 512 MiB write to 256 MiB of temp space, with the write slowing from 129 to 107 MB/s. ### Tests The harness in a throwaway Debian 13 container passes 46 of 46. That's the original 11 scenarios plus 17 new ones, each with one or more checks. - **12:** a name mapped to a public test address prints the warning, naming 192.168.0.53. A name mapped to a private address mounts with no warning. - **13 and 14:** the defaults by CPU count. With 8 CPUs, `-o` replaces a default in place, and a new mount on one CPU gets `multipart_size=64` only, since an existing mount now keeps its settings. - **15:** `--list`, as root and as an ordinary user, shows all six mounts, `--no-fstab` ones included. Naming a bucket mounted in several places asks for the mount point. - **16–17:** `--set` and `--unset` change the fstab line in place and remount. The disk cache fills on a read. - **18:** an option s3fs refuses (`multipart_size=1`) is rolled back, and the bucket stays mounted. - **19:** settings that need more temp space than there is are refused before anything changes. - **20:** running the deploy again keeps edited settings. - **21:** a `--no-fstab` mount is changed by remounting it, and stays out of fstab. - **22–23:** a RAM-disk temp directory, and stream uploads on it, produce the tips. A new mount on a machine whose `/tmp` is a RAM disk gets `tmpdir=/var/tmp/s3fs`. - **24–25:** the menu, driven from a terminal: the guidance is shown, changes are marked, and saving remounts. Leaving without saving changes nothing. - **26:** `--remove` unmounts and removes the line, and keeps credentials another mount still uses. - **27:** `--list`, `--edit` and the menu go through `--pct`, with a stand-in for `pct`, and the copy pushed into the container is cleaned up. - **28:** `--set` values that would break fstab are refused. The script is shellcheck-clean at warning level. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
A Jellyfin container's S3 reads crawled, and a plain read over the same
path failed outright: its DNS was the router's and 1.1.1.1, as the
script's own advice said to set, and those only have the internet's
answer for the endpoint, so every request went out to the public edge and
back. Pointed at the LAN's DNS (192.168.0.53, .54), which answers with
Junction's local addresses, it read 1 GiB at 143 MiB/s.

- the advice for a name that does not resolve names the LAN's DNS
- a name that resolves only to public addresses gets a warning naming
  it (a machine really outside the network wants the public path, so it
  does not fail)
- mounts default to parallel_count=16, multipart_size=64,
  multireq_max=32 and max_background=64: on a 1 GiB file through
  Junction, 259 -> 308 MiB/s read start to end, ten 4 MiB seeks in 2.1 s
  instead of 3.2; -o still overrides them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured further, on 1 GiB written with fdatasync and read back through
Junction: s3fs is CPU-bound (it hashes and encrypts every part), about
80-90 MB/s on one CPU whatever the options. The parallel options help
only with CPUs to spare - on one CPU they slowed reads from 129 to 82
MB/s - so they are added only from 4 CPUs; multipart_size=64 helps a
little everywhere and stays. streamupload doubles writes with CPUs to
spare (133 -> 184 MB/s on 2, 159 -> 275 on 8) but s3fs still calls it
experimental, so the README offers it with -o rather than defaulting to
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run with no arguments on a machine that has s3fs mounts (in /etc/fstab, or
mounted with --no-fstab), the script lists them and opens a menu of a mount's
settings: caching, speed, temp space, access and timeouts. Each line shows its
value and its trade-off in a few words; choosing one explains what it does and
what raising it costs before asking. Tips above the menu name what this
machine gets wrong: a temp directory on a RAM disk, more parallel transfers
than its CPUs can encrypt, settings s3fs would refuse to mount with. Saving
rewrites the fstab line in place and remounts; if s3fs refuses, the old line
goes back and the bucket is mounted as it was. --list, --edit with
--set/--unset/-o, and --remove do the same without the menu, and all of it
works through --pct.

The guidance is measured with s3fs 1.95, not assumed:
- s3fs itself stays at 10-13 MiB whatever the part size or parallel count
- it holds every open file whole in its temp directory, read or written,
  so a RAM-disk /tmp holds them in memory: new mounts on such a machine
  use /var/tmp/s3fs instead
- it won't mount unless its temp disk has part size x parallel transfers
  free over its 10% floor; the defaults now fit, and --edit checks first
- stream uploads fail a write ("No space left on device") when the temp
  directory fills, where s3fs otherwise slows down; the README no longer
  says they avoid the copy into /tmp
- max_dirty_data=256 kept a 512 MiB write to 256 MiB of temp space

Running the deploy again keeps a mount's settings (defaults, then the
existing line, then the script's own options, then -o), so it no longer
undoes changes made with --edit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mc-dev-2b8ea366 changed title from s3fs-deploy.sh: the LAN DNS, a warning for a public answer, and faster defaults to s3fs-deploy.sh: LAN DNS, faster defaults, and a menu to change a mount's settings 2026-10-31 10:08:37 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jacob/tftp!1
No description provided.