[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jgit-dev] DFS reftable at Google

So I posted two long topics, reftable and dfs-reftable, because Google
is moving to reftable to back our DFS reference storage.

Due to hilarious mistakes I made ~7 years ago, our internal database
system is no longer suitable for storing huge numbers of references
(e.g. 866k) in Gerrit Code Review repositories. We could fix this and
keep using a Google database, but it doesn't solve all of the problems
that reftable solves. Or we could build reftable, include it in JGit,
and solve a lot of problems. I chose the latter. :)

One of the great features of reftable is readers will see atomic
updates of multi-ref batches. This is great for Gerrit Code Review
when a patch set is both added, and the /meta ref is updated. Readers
won't have any risk of seeing a partial state.

I'm hoping I can get git-core to also adopt reftable, such that Gerrit
Code Review can eventually recommend admins use reftable instead of
$GIT_DIR/{refs,packed-refs} for reference storage. But that is still a
work in progress.

No matter what happens in git-core, Google is moving its system to
reftable, and will be maintaining reftable within DFS.

If git-core rejects reftable, Gerrit Code Review may support reftable
as an option, but we'll continue to encourage Gerrit to be compatible
with git-core, meaning it stays with $GIT_DIR/{refs,packed-refs}. Or
switches to some other format that git-core moves to, such as LMDB.

Ketch's RefTree may be a bad idea in the long run. It doesn't scale
with references the way reftable does. I've got an unpublished draft
document how to efficiently embed reftable within a canonical tree
structure, to replace RefTree for Ketch.

I haven't published it yet because Ketch is very much a back-burner
project to us, and reftable needs to land and be stable before the
Ketch prototype could really start to depend on an embedded reftable
for the consensus system.