Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

Most conversations about AI sandboxes start and end with boot speed and compute isolation: How fast does the container boot? Can the agent run untrusted code safely? But these discussions miss a fundamental truth: an autonomous agent turns a fresh machine into a rambling digital workspace. It clones code repos, installs heavy node_modules or Python virtual environments, downloads browser binaries, compiles artifacts, generates multi-gigabyte datasets, and writes massive trace logs.

"The files are not incidental. They are the working memory of the task."
– Aniket Maurya, Founder @ Celesto AI

The naive solution is to overprovision the root disk. However, determining the size of a boot disk before a task exposes its scope is an ineffective effort. A "quick bug fix" might require a full Next.js build chain, while a "data benchmark" might produce multi-gigabyte Parquet tables. Coupling system state to workspace state creates a bottleneck.

Instead, an emerging architectural pattern is radical in its simplicity: decouple the system disk from the durable workspace. Provide a POSIX-compliant mount that acts like an infinite, persistent drive, backed entirely by affordable object storage. But how do you build such a system without introducing complex services that defeat the purpose of ephemeral sandboxes? Recently, we co-hosted an Office Hours session with Celesto AI, a platform building secure sandboxes for autonomous AI agents, and the webinar was a delightful exploration of exactly this challenge.

Here is the creative engineering stack making this possible: JuiceFS as the file system layer, SQLite as the embedded metadata engine, and Litestream as the durability glue.

The creative trio: JuiceFS + SQLite + Litestream

Object stores (S3, GCS) are excellent for storing files and performing key-based lookups, but they are terrible at listing directories, handling stat calls, or managing millions of small files in batches (i.e., renaming a directory). You need a metadata engine to act as the "brain" of the file system, mapping paths to block locations.

Usually, JuiceFS deployments use centralized or clustered metadata services, such as Redis, TiKV, or FoundationDB. For isolated AI sandboxes, a creative alternative leverages the three familiar JuiceFS components working in harmony:

1. The JuiceFS client provides the POSIX bridge. It is responsible for the complex task of communicating with both the metadata engine and the object store. It is the "worker" that reads and writes the actual data, as well as performs administrative tasks such as status checks and data compaction.

2. SQLite acts as the metadata engine. Instead of connecting to a remote metadata server, JuiceFS also supports using a local SQLite database file as its metadata store. This means the entire file system metadata (every inode, directory entry, and block pointer) lives inside a single, standard .db file on the local disk.

3. Litestream provides the durability backing for that SQLite file. Litestream continuously takes SQLite snapshots and streams the SQLite write-ahead log (WAL) frames to the same object store.

Here is the magic: When the sandbox is running, JuiceFS uses the local SQLite file to resolve ls, cd, and read operations. The actual file blocks are already safe in the object store. (Although, the sync-interval configuration needs to be taken into consideration.) When the user stops the sandbox, the local SQLite file might be discarded, but Litestream has already replicated the metadata engine to the object store.

When the sandbox resumes on a fresh VM (or a different host entirely), the system simply downloads the latest SQLite snapshot from the object store and replays the WAL. Within seconds, the local SQLite can be fully reconstructed. JuiceFS mounts it, points back to the same object store blocks, and the agent is looking at the exact same workspace directory it left behind.

The JuiceFS file system doesn't care which computer it runs on. But it does remember its folder structure and file pointers with the portable SQLite snapshots and WAL frames. With this setup, we turned the working memory of the AI agent into files and blocks in the object store, making it durable and accessible instead of locking it to a specific boot disk or block device.

An unconventional but effective combo

JuiceFS supports interchangeable metadata engines. Choosing SQLite + Litestream subverts the typical playbook by prioritizing simplicity and per-agent isolation over global scale. Here is why this particular stack fits the AI agent use case so well:

• Easy to set up: While it is not completely hands-off (you still manage the JuiceFS client, the local SQLite instance, and Litestream's replication configuration like Celesto AI does), this stack is drastically simpler to bootstrap than a distributed database cluster.

• Per-agent isolation: Each sandbox gets its own dedicated SQLite metadata file. This eliminates noisy-neighbor problems. Metadata contention simply does not happen across agents, because each one has its own independent "brain" living right next to its mount.

• Performance that fits the scope: A single agent workload typically produces tens of thousands to a few million files. SQLite handles this range comfortably with low latency and minimal resource footprint. While this approach obviously does not scale to the hundreds of billions of files that JuiceFS Enterprise Edition can manage globally, it falls squarely into the comfortable zone for individual agent workspaces.

Of course, this approach comes with important trade-offs and considerations:

• No cross-agent sharing: The file system is inherently tied to a single SQLite instance. Agents cannot mount and share the same workspace concurrently. If your use case requires collaborative access or shared datasets across sandboxes, you would need a different strategy.

• Resume latency depends on object store speed: When a sandbox resumes, the system must download the latest SQLite snapshot from the object store and replay the WAL frames. This adds noticeable startup time compared to a hot-mounted local disk. Tuning Litestream's sync-interval involves a direct trade-off between durability guarantees and resume performance.

• Single-writer limitation: SQLite is fundamentally a single-writer database. Heavy parallel writes from multiple processes within the same sandbox can become a bottleneck. For autonomous agents following a largely sequential workflow, this limitation is rarely a practical issue. But it is worth keeping in mind for highly concurrent, multi-threaded agent designs.

The stack is not a universal solution. It is a deliberate, opinionated choice that trades global scalability and shared access for extreme simplicity and tight per-agent isolation. And for the AI sandbox use case, that trade-off often makes perfect sense.

The performance reality

This architecture is beautiful, but it isn't magic. You are trading local SSD latency for object storage durability and infinite scale. The benchmarks reveal a clear pattern:

• Small files need attention: Writing 5,000 tiny 4 KiB files to an object-backed file system involves heavy metadata churn. Even with SQLite's fast local access, every close() and fsync() triggers WAL writes and Litestream replication. In these scenarios, the local root disk (ext4/XFS) will outperform the durable mount by a factor of 2x to 5x. See Celesto's blog post for more details.

• Large sequential workloads shine: For datasets, model checkpoints, build artifacts, and logs (files > 10 MB), the object-backed mount achieves upwards of 200 MiB/s, which is nearly 90% of the performance of the local disk. The background block uploads are streamed efficiently. See Celesto's blog post for more details.

The golden rule for agents is to use the object-backed mount drive for durable, long-term state reports and large artifacts. Use the local ephemeral disk for high-churn, temporary scratch work that doesn't need to survive a restart. This hybrid approach gives you infinite capacity without sacrificing performance where it truly hurts.

Final thoughts

Cloudflare CFO Thomas Seifert recently predicted that within five years, non-human traffic could be as much as 1,000 times human traffic, making humans "a rounding error on the internet." He caveated that he's "called it wrong at every point along the way," but the direction is hard to dispute: AI and machine traffic already overtook human traffic in May 2026.

If that is true for network packets, it is likely true for storage as well.

Every autonomous agent session produces a workspace. Multiply that by millions of agents running continuously, resuming, retrying, and branching. Human developers generate storage in bursts tied to working hours. Agents generate it constantly, at machine speed.

This raises an uncomfortable question: if non-human consumers are about to dominate storage demand, are we designing for the right consumer? The combo of JuiceFS + SQLite + Litestream is one answer, but not the only one. Other approaches worth watching include object storage as the primary interface, shared metadata engine with concurrent access at agent scale (i.e., the JuiceFS Enterprise Edition), copy-on-write workspace snapshots, and many others.

Portable, shareable, scalable file storage for AI agents is still in its opening moves.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.