A reference lakeFS target

The parts

Two programs run side by side. Both are built from the fork rahuldave/lakeFS, branch provcat.

  provcat (acme's keys)        provcat (globex's keys)
           |                            |
           v                            v
  +------------------------------------------------+
  | lakeFS 1.86.0, patched        127.0.0.1:8002   |
  |   S3 gateway and REST API                      |
  |   data, commits and branches in <dir>/data     |
  +------------------------------------------------+
           |  "may this user do this?"  (Bearer token)
           v
  +------------------------------------------------+
  | access-list server, patched   127.0.0.1:8011   |
  |   users, groups, policies and keys             |
  +------------------------------------------------+

• lakeFS holds the data, the commits and the branches. It runs with auth.ui_config.rbac: external, so it asks another server who may do what.
• The access-list server is the reference authorization server that lakeFS ships in contrib/auth/acl. It keeps the users, groups, policies and keys. lakeFS fetches each user’s policies from it, and checks every call against them.
• packages/provcat/scripts/lakefs_acl_local.py in this project starts both, sets up the orgs, and stops both.

Start and stop it

Build the fork first, as the setup page explains under “Build lakeFS 1.86.0 from source”. Then start the server:

uv run python packages/provcat/scripts/lakefs_acl_local.py

It runs in the foreground. If you press Ctrl+C, run kill or close the terminal, lakeFS stops, and then the access-list server stops. Start it again with the same command.

When it is ready, it prints two lines:

lakeFS on http://127.0.0.1:8002, with access control for acme, globex
export AWS_SHARED_CREDENTIALS_FILE=/Users/<you>/.local/share/provcat/lakefs-acl/credentials

The script has these options:

The first run fixes the ports, because the config files hold them. The script refuses a later run with other ports. It also refuses a port that something else already uses.

The folder

The first run makes the folder and writes everything into it. Later runs reuse it.

To start over, stop the server and delete the folder. The next run makes new keys for everyone.

The script passes four settings to the programs in the environment, not in the config files. So an older folder gets them too.

Orgs, users and keys

Each org gets four things:

• A group, named after the org, for example acme.
• A user, <org>-user, in that group.
• Keys for that user, saved in credentials as the profile lakefs-<org>.
• A policy, <org>-repos, attached to the group.

The policy for acme is this:

{"id": "acme-repos", "statement": [
  {"effect": "allow", "action": ["fs:*"],
   "resource": "arn:lakefs:fs:::repository/acme-*"},
  {"effect": "allow", "action": ["fs:AttachStorageNamespace"],
   "resource": "arn:lakefs:fs:::namespace/local://acme-*"}
]}

So acme can do anything in a repository whose name starts with acme-, and can make one only on storage named local://acme-.... An org name is 1 to 20 of a-z and 0-9, with no -. So acme-* never matches another org’s repositories.

The admin is the user admin, in the group Admins. Its keys are the profile lakefs. The admin can do everything, in every repository.

uv run python packages/provcat/scripts/lakefs_acl_local.py --org acme --org globex --org initech

Use it with provcat

Point provcat at the server and its keys:

export AWS_SHARED_CREDENTIALS_FILE=~/.local/share/provcat/lakefs-acl/credentials

export LAKEFS_S3_ENDPOINT=http://127.0.0.1:8002

The example configuration, packages/provcat/config/provcat.example.yaml, then works as it is. Its store lakefs-acme uses the profile lakefs-acme, and lakefs-globex uses lakefs-globex. The logical schemes pick the store from the org in the URI, so data+table://acme/... reads and writes with acme’s keys only. The store lakefs uses the admin’s keys.

Give each org a configuration with only its own store, and only its own profile in its credentials file. The mapping in one shared configuration routes requests. The keys keep orgs apart. The design explains this under “Tenants”.

AWS_SHARED_CREDENTIALS_FILE hides ~/.aws/credentials in that shell. So the plain S3 stores lose their keys there. Use another shell for them, or copy the profiles into ~/.aws/credentials.

Run it as the admin

The admin manages the server through the lakeFS REST API, with the keys of the profile lakefs. lakeFS passes the calls about users, groups and policies on to the access-list server. These calls are the useful ones:

For example, with curl and the admin keys from the profile:

curl -u "$ADMIN_KEY:$ADMIN_SECRET" http://127.0.0.1:8002/api/v1/auth/users

An org’s user cannot list, make or delete its own keys. Only the admin can.

A change to a policy can take up to about 23 seconds to apply, because lakeFS caches each user’s policies.

uv run python packages/provcat/scripts/lakefs_acl_local.py --repair-admin

What the setup protects and what it does not

The setup protects each org’s data from the other orgs:

• Org keys. An org’s keys reach only that org’s repositories. That covers objects, commits, branches, staged and linked addresses, storage names and meta-range IDs.
• The access-list server. It takes only calls that carry the token, and it listens on 127.0.0.1 only.
• The secrets. The config files, the token and the keys are readable by your user only.

The setup does not protect against these:

• Your own user account. Anyone who can read <dir> has the admin keys and the token.
• Other machines. Both programs listen on 127.0.0.1. Do not expose them on a network without TLS and a firewall.
• Revocation at once. lakeFS caches each user’s policies, so a revoked policy can work for up to about 23 seconds.
• Server paths. Error messages and stat replies show absolute paths on the server.
• Every admin mistake. The guards cover the admin’s own policy, and --repair-admin undoes the rest. They do not stop the admin from deleting an org’s data.
• A rare race. If a policy is deleted and made again at the same moment that it is attached, a stale link can survive. It has not been reproduced.

Test it

Integration tests run the script against fake servers, in about 40 seconds:

uv run pytest packages/provcat/tests/integration/test_lakefs_acl_local.py

The system tests run against the real server. Start it first, then:

PROVCAT_SYSTEM_LAKEFS_ACL=http://127.0.0.1:8002 \
PROVCAT_SYSTEM_LAKEFS_ACL_SERVER=http://127.0.0.1:8011 \
PROVCAT_SYSTEM_LAKEFS_ACL_DIR=~/.local/share/provcat/lakefs-acl \
PROVCAT_SYSTEM_LAKEFS_ACL_CREDENTIALS=~/.local/share/provcat/lakefs-acl/credentials \
PROVCAT_SYSTEM_LAKEFS_ACL_DATA=~/.local/share/provcat/lakefs-acl/data \
uv run pytest -m system packages/provcat/tests/system/test_lakefs_acl.py

They take about 3 minutes, most of it waiting out lakeFS’s policy cache after each repair. They make repositories named acme-provcat-test-... and globex-provcat-test-..., and delete them at the end. The repair tests lock the admin out and repair it, so they change the admin’s keys in credentials.

The fork has its own Go tests. From ~/Projects/lakeFS:

GOTOOLCHAIN=go1.26.4 go test ./pkg/api ./pkg/graveler/... ./pkg/catalog/... ./pkg/pyramid/... ./contrib/auth/...

Errors and their fixes

The lakeFS banner says to complete setup at 127.0.0.1:8000. That line is stock lakeFS text and does not apply. The script has set everything up.

Where to read more

• The fork’s changes: PROVCAT.md at the root of rahuldave/lakeFS.
• Building lakeFS: the section “Build lakeFS 1.86.0 from source” in the setup page.
• Tenancy: the section “Tenants” in the design, with the holes in stock lakeFS.
• Writes through provcat: the section “Writes” in the design.
• The script itself: packages/provcat/scripts/lakefs_acl_local.py.