# Large monorepos

Index a multi-million-line monorepo with its full git history on one machine, from the first clone to keeping the index current on every commit.

repowise indexes a multi-million-line monorepo on a single machine, with its
full git history. This page gives the commands for a repository of a few
million lines and tens of thousands of commits, from the first clone to keeping
the index current.

A machine with 8 vCPUs, 32 GB of RAM and 50 GB of free disk indexes a
4-million-line repository with its full history; give it 64 GB of RAM when
PostgreSQL runs on the same machine. The first index is a one-time cost. After
it, `repowise update` re-analyses only what each commit changed, and the
health walk, dead code and deep git history run in parallel worker processes.

## 1. Prepare the checkout

Index a normal clone with its history:

```bash
git clone https://gitlab.example.com/group/monorepo.git
cd monorepo
git rev-list --count HEAD        # how many commits you have
```

Use a full clone and choose how much history to read with `--commit-limit`
(below), rather than trimming the clone. A shallow clone (`--depth` or
`--shallow-since`) leaves cut-off commits at its boundary, and git reports each
of them as adding every file in the repository, which slows the history pass.
If you already have a shallow clone, run `git fetch --unshallow` before the
first index.

**C and C++ include paths.** repowise resolves `#include` lines through
`compile_commands.json` when it finds one at the repo root or in `build/`,
alongside the CMake and Bazel target files it reads on its own. A CMake
configure step is enough to write it; nothing has to compile:

```bash
cmake -S . -B build -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
```

For Bazel, generate it with a compile-commands extractor and leave the file at
the repo root. Python and Scala need no setup: Python imports resolve from the
tree, and Scala resolves through the shared JVM index, reading SBT and Mill
build files where it needs them.

**Exclusions.** Leave out what nobody edits: vendored libraries, generated
code, large test fixtures. Put the patterns in `.repowiseIgnore` at the root
(gitignore syntax, read on every run) or pass them once with `-x`:

```bash
cat > .repowiseIgnore <<'EOF'
third_party/
contrib/
**/generated/
tests/fixtures/data/
EOF
```

Git submodules are skipped by default; `--include-submodules` indexes them.

## 2. Choose the database

| | SQLite (default) | PostgreSQL |
|---|---|---|
| Best for | One machine, one user | A shared server, the web UI and MCP for a team |
| Setup | Nothing: `.repowise/` in the repo | PostgreSQL with the `pgvector` extension available |

For PostgreSQL:

```bash
pip install "repowise[postgres]"
export REPOWISE_DB_URL="postgresql+asyncpg://repowise:secret@db.internal:5432/repowise"
```

The schema is created on first use. After the first index on PostgreSQL, run
`ANALYZE;` once in that database so the query planner sees the new table
sizes. For long-lived connections through a firewall,
`REPOWISE_POSTGRES_KEEPALIVE=1` turns on keepalive probes and a command
timeout ([Environment variables](https://docs.repowise.dev/installation/env-vars#server-and-database)).

## 3. First index

```bash
REPOWISE_PARSE_WORKERS=4 repowise init --yes --no-prose --no-editor-setup \
  --commit-limit 25000
```

| Flag | Why |
|---|---|
| `--no-prose` | Builds every page from code, with no model and no key. Add the model-written concept pages later with `repowise generate`. |
| `--commit-limit 25000` | Reads the newest 25,000 commits for file history, ownership, co-change and the commit list. Set it above `git rev-list --count HEAD` to read the full history. Saved to config. |
| `REPOWISE_PARSE_WORKERS=4` | Parse processes. Each holds its own parser, so this is the main memory lever; the default is the CPU count, at most 8. |
| `--no-editor-setup` | On a server, writes nothing outside `.repowise/`. Drop it on a developer machine to wire up editors and agents. |
| `--yes` | Takes every default without asking. |

For the fastest first pass, index the graph and essential history first and
add the rest later, with no model calls either way:

```bash
repowise init --yes --no-prose --mode fast
repowise update --full
```

An interrupted first index continues where it stopped with
`repowise init --resume`.

### Full history

`commit_limit` sets how far back churn, ownership, co-change and the commit
list see; Change Records cover the newest 10,000. The default is the newest
5,000 commits, there is no upper bound, and depth is cheap: deep history is
read four regions at a time with git's pack memory bounded. On git/git (61,453
non-merge commits), reading the full history instead of 5,000 commits adds a
few seconds and about 100 MB to the index.

To deepen an existing index without re-indexing:

```bash
repowise config set commit_limit 25000
repowise update --index-only --defer-docs --refresh-git
```

### Files left out

Source files up to 8 MB are parsed: those over 2 MB one at a time, outside the
parallel worker pool, so memory stays bounded. Source files over 8 MB,
minified bundles, files with no parser over 500 KB, and generated files are
left out of the index. The console names the
first few source files; every one of them, with its size and reason, is in
`.repowise/skipped_files.txt`, rewritten on each `init` and `update`:

```text
# Files repowise did not index: path, size, reason (tab-separated)
src/parser/grammar_tables.cpp	9,310 KB	over the 8,192 KB limit
tests/data/fixtures.tsv	48,112 KB	over the size limit for files without a parser
```

## 4. Keep it current

`repowise update` diffs the new commit against the last indexed one and
re-analyses only what changed. Pick a trigger ([Auto-sync](https://docs.repowise.dev/auto-sync)
has the details):

| Setup | Trigger |
|---|---|
| Developer machine | The post-commit hook `init` installs (`repowise hook install`) |
| Shared server on GitLab | The GitLab push webhook, with the 15-minute poller as a safety net |
| Server-owned clone | `managed_checkout: true`, so each sync fetches and fast-forwards the clone first |

Source edits, added and removed files, documentation and config edits patch
the graph in place, C and C++ included: on a 62,000-file C++ repository an
update takes 52 s where a full graph rebuild takes 303 s. A change to a build
file the graph reads (`CMakeLists.txt`, `BUILD`, `BUILD.bazel`) in a way that
changes the build rebuilds the whole graph on that update, because include and
target resolution depend on it.

## Memory and time levers

In order of effect:

```bash
REPOWISE_PARSE_WORKERS=2 repowise init ...   # fewer parse processes, less memory
repowise init --mode fast ...                # index only; `repowise update --full` adds the rest
repowise init -x 'third_party/' ...          # exclude what you never edit
repowise init --max-file-pages 2000 ...      # cap file pages, highest importance first
```

After the first index, `repowise config list --suggested` reads what it stored
and names the settings that fit this repository, each with its numbers and the
command: a deeper `commit_limit` when many files have no commit in the window,
`exclude_patterns` for directories holding only vendored or generated code,
and every file page when the automatic cap left production files out.

To index the monorepo together with its smaller sibling repositories behind
one MCP server, see [Workspace setup](https://docs.repowise.dev/multi-repo/workspace-setup).
