Setting-Up-Storage.md

Setting Up Storage

ArchiveBox supports a wide range of local and remote filesystems using rclone and/or Docker storage plugins. The examples below use Docker Compose bind mounts to demonstrate the concepts; adapt the host paths, ownership, and provider settings to your environment.

Example docker-compose.yml storage setup:

services:
    archivebox:
        # ...other service settings...
        volumes:
            # your index db, config, logs, etc. should be stored on a local SSD (usually <10Gb)
            - ./data:/data

            # but bulk archive/ content can be located on an HDD or remote filesystem
            - /mnt/archivebox-archive:/data/archive

Related Docs

• README: Archive Layout
• Wiki: Usage (Disk Layout)
• Wiki: Usage (Large Archives)
• Wiki: Security Overview (Output Folder)
• Wiki: Publishing Your Archive
• Wiki: Upgrading or Merging Archives
• Wiki: Troubleshooting Filesystem Issues

---

Supported Local Filesystems

EXT4 (default on Linux), APFS (default on macOS)

Tip

These default filesystems are fully supported by ArchiveBox on Linux and macOS (w/wo Docker).

ZFS (recommended for experienced Linux/BSD operators) ⭐️

Tip

ZFS is a good choice when you already operate OpenZFS and want checksumming, compression, snapshots, replication, and optional encryption or disk redundancy. On Ubuntu, follow the official OpenZFS installation guide. macOS and BSD installation and property support differ, so use the guide for your OS.

• https://openzfs.github.io/openzfs-docs/
• https://openzfs.github.io/openzfs-docs/man/v2.2/8/zpool-create.8.html
• https://openzfs.github.io/openzfs-docs/man/v2.2/8/zfs-create.8.html
• https://docs.docker.com/storage/storagedriver/zfs-driver/
• https://www.ixsystems.com/blog/fast-dedup-is-a-valentines-gift-to-the-openzfs-and-truenas-communities/

Caution

Creating a pool erases the selected disks. The two-disk example below creates a mirror; replace both /dev/disk/by-id/... placeholders with the persistent IDs of empty disks you intend to erase.

# Create a mirrored pool without forcing ZFS's safety checks.
sudo zpool create \
    -O mountpoint=none \
    -O compression=lz4 \
    -O dnodesize=auto \
    -O atime=off \
    -O xattr=sa \
    -O acltype=posixacl \
    -O aclinherit=passthrough \
    archivebox mirror \
    /dev/disk/by-id/DISK_ONE \
    /dev/disk/by-id/DISK_TWO

# Create the unencrypted ArchiveBox data dataset.
sudo zfs create \
    -o mountpoint=/mnt/archivebox/data \
    archivebox/data

To encrypt a new dataset, use this command instead of the unencrypted zfs create command above. ZFS encryption must be selected when the dataset is created.

sudo zfs create \
    -o mountpoint=/mnt/archivebox/data \
    -o encryption=on \
    -o keyformat=passphrase \
    -o keylocation=prompt \
    archivebox/data

NTFS, HFS+, BTRFS

Warning

These filesystems are likely supported, but are not officially tested.

EXT2, EXT3, FAT32, exFAT

Caution

Not recommended. Cannot store files >4GB or more than 31k ~ 65k Snapshot entries due to directory entry limits.

---

Supported Remote Filesystems

ArchiveBox supports many common types of remote filesystems using Rclone, FUSE, Docker storage providers, and Docker volume plugins.

The data/archive/ subfolder contains the bulk archived content, and it supports being stored on a slower remote server (SMB/NFS/SFTP/etc.) or object store (S3/B2/R2/etc.). For data integrity and performance reasons, the rest of the data/ directory (data/ArchiveBox.conf, data/logs, etc.) must be stored locally while ArchiveBox is running.

Important

data/index.sqlite3 is your main archive DB, it must be on a fast, reliable, local filesystem which supports FSYNC (SSD/NVMe recommended for best experience).

Tip

If you use a remote filesystem, you should switch ArchiveBox's search backend from ripgrep to sonic (or FTS5).
_{(ripgrep scans over every byte in the archive to do each search, which is slow and potentially costly on remote cloud storage)}

NFS (Docker Driver)

docker-compose.yml:

services:
    archivebox:
        volumes:
            - ./data:/data
            - archivebox-archive:/data/archive

volumes:
    archivebox-archive:
        driver: local
        driver_opts:
            type: "nfs"
            o: "addr=some-remote-server.example.com,rw,nfsvers=4"
            device: ":/archivebox-archive"

SMB / Ceph (Docker CIFS Driver)

docker-compose.yml:

services:
    archivebox:
        volumes:
            - ./data:/data
            - archivebox-archive:/data/archive

volumes:
    archivebox-archive:
        driver: local
        driver_opts:
            type: cifs
            device: "//some-remote-server.example.com/archivebox-archive"
            o: "username=XXX,password=YYY,uid=911,gid=911"

Amazon S3 / Backblaze B2 / Google Drive / etc. (RClone)

ArchiveBox stores snapshot content under data/archive/users/<user>/snapshots/<date>/<domain>/<uuid>/ and keeps backwards-compatible data/archive/<timestamp> symlinks. Object-storage mounts must enable Rclone's VFS symlink translation so both parts of this layout work.

Install the rclone binary through abxpkg:

uv tool install abxpkg
abxpkg install rclone
abxpkg run rclone version

Then install the FUSE 3 system integration supplied by your OS. For example, on Ubuntu:

sudo apt-get install fuse3
grep -qxF user_allow_other /etc/fuse.conf ||
    printf '%s\n' user_allow_other | sudo tee -a /etc/fuse.conf

Then define your remote storage config ~/.config/rclone/rclone.conf:

Tip

You can also create rclone.conf using the Rclone Web GUI: abxpkg run rclone rcd --rc-web-gui

# Example rclone.conf using Amazon S3 for storage:
[archivebox-s3]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = YYY
region = us-east-1

Rclone Config Examples

• SMB / Ceph / SFTP / FTP / WebDAV (e.g. Nextcloud)
• Google Drive / Dropbox / OneDrive
• Amazon S3 / Backblaze B2 / Cloudflare R2 / DigitalOcean Spaces
• Google Cloud Storage / Azure Blob / Azure Files
• Storj / Sia / Archive.org Storage
• And many more...
  • https://rclone.org/s3/
  • https://rclone.org/overview/

Bonus:

• Set up gzip compression: https://rclone.org/compress/
• Set up file encryption: https://rclone.org/crypt/
• Set up hashing engine: https://rclone.org/hasher/

Option A: Running Rclone on a bare-metal host

1. If Needed: Transfer any existing local archive data to the remote volume first

Caution

Stop ArchiveBox before migrating its archive directory. rclone sync makes the remote destination match the local source and can delete files already present at the destination. Run it with --dry-run first, make a separate backup, and do not move the local copy until rclone check succeeds.

abxpkg run rclone sync \
    --dry-run \
    --links \
    --fast-list \
    --transfers 20 \
    --progress \
    /opt/archivebox/data/archive/ \
    archivebox-s3:data/archive/

# Remove --dry-run only after reviewing the proposed changes, then verify them.
abxpkg run rclone sync \
    --links \
    --fast-list \
    --transfers 20 \
    --progress \
    /opt/archivebox/data/archive/ \
    archivebox-s3:data/archive/
abxpkg run rclone check --links /opt/archivebox/data/archive/ archivebox-s3:data/archive/

mv /opt/archivebox/data/archive /opt/archivebox/data/archive.localbackup
mkdir -p /opt/archivebox/data/archive

2. Mount the remote storage volume as FUSE filesystem

Run the mount as the numeric user that owns the local ArchiveBox collection. The command stays in the foreground so a service manager can supervise it.

abxpkg run rclone mount \
    archivebox-s3:data/archive/ \
    /opt/archivebox/data/archive/ \
    --allow-other \
    --vfs-cache-mode=full \
    --vfs-links \
    --transfers=16 \
    --checkers=4

See Rclone's rclone mount documentation for service-manager and cache-size configuration.

Tip

You can use an existing Rclone FUSE mount as a normal Docker bind mount. A separate storage plugin is usually unnecessary when user_allow_other and --allow-other are configured correctly.

docker run --rm \
    -v "$PWD:/data" \
    -v /opt/archivebox/data/archive:/data/archive \
    archivebox/archivebox:dev status

docker-compose.yml:

services:
    archivebox:
        # ...other service settings...
        volumes:
            - ./data:/data
            - /opt/archivebox/data/archive:/data/archive

Option B: Running Rclone with the Docker storage plugin

This Linux Docker Engine option is only needed if you cannot use Option A for compatibility or performance reasons, or if you prefer defining your remote storage in docker-compose.yml.

See here for full instructions: Rclone Documentation: Docker Plugin

1. First, install the Rclone Docker Volume Plugin for your CPU architecture (e.g. amd64 or arm64):

sudo mkdir -p \
    /var/lib/docker-plugins/rclone/config \
    /var/lib/docker-plugins/rclone/cache
sudo install -m 600 \
    ~/.config/rclone/rclone.conf \
    /var/lib/docker-plugins/rclone/config/rclone.conf

# Replace amd64 with arm64 on ARM hosts.
docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rclone

2. Then, create a volume using the Docker CLI or define one using Docker Compose / Swarm:

docker-compose.yml:

services:
    archivebox:
        volumes:
            - ./data:/data
            - archivebox-s3:/data/archive

volumes:
    archivebox-s3:
        driver: rclone
        driver_opts:
            remote: 'archivebox-s3:data/archive'
            allow_other: 'true'
            vfs_cache_mode: full
            vfs_links: 'true'
            # Match these to the numeric owner of ./data; 911:911 is the image default.
            uid: 911
            gid: 911
            transfers: 16
            checkers: 4

To start the container and verify the filesystem is accessible within it:

docker compose run --rm archivebox \
    /bin/bash -c 'touch /data/archive/.write_test && rm /data/archive/.write_test'

---

More Docker Storage Plugins

• IPFS / Peergos / GlusterFS
• DigitalOcean Block Storage Volumes / Linode Block Storage Volumes
• More volume plugins...