Dturner/pluggable backends #2
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dturner-tw
force-pushed
the
dturner/pluggable-backends
branch
from
68e0e00 to
4665727
Compare
dturner-tw
pushed a commit
that referenced
this pull request
Jun 25, 2015
The collect_parents() function now is responsible for
1. parsing the commits given on the command line into a list of
commits to be merged;
2. filtering these parents into independent ones; and
3. optionally calling fmt_merge_msg() via prepare_merge_message()
to prepare an auto-generated merge log message, using fake
contents that FETCH_HEAD would have had if these commits were
fetched from the current repository with "git pull . $args..."
Make "git merge FETCH_HEAD" to be the same as the traditional
git merge "$(git fmt-merge-msg <.git/FETCH_HEAD)" $commits
invocation of the command in "git pull", where $commits are the ones
that appear in FETCH_HEAD that are not marked as not-for-merge, by
making it do a bit more, specifically:
- noticing "FETCH_HEAD" is the only "commit" on the command line
and picking the commits that are not marked as not-for-merge as
the list of commits to be merged (substitute for step #1 above);
- letting the resulting list fed to step #2 above;
- doing the step #3 above, using the contents of the FETCH_HEAD
instead of fake contents crafted from the list of commits parsed
in the step #1 above.
Note that this changes the semantics. "git merge FETCH_HEAD" has
always behaved as if the first commit in the FETCH_HEAD file were
directly specified on the command line, creating a two-way merge
whose auto-generated merge log said "merge commit xyz". With this
change, if the previous fetch was to grab multiple branches (e.g.
"git fetch $there topic-a topic-b"), the new world order is to
create an octopus, behaving as if "git pull $there topic-a topic-b"
were run. This is a deliberate change to make that happen, and
can be seen in the changes to t3033 tests.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
dturner-tw
pushed a commit
that referenced
this pull request
Jun 25, 2015
The controlling tty-based heuristics to squelch progress output did not consider that the process may not be talking to a tty at all (e.g. sending the progress to sideband #2). This is a finishing touch to a topic that is already in 'master'. * lm/squelch-bg-progress: progress: treat "no terminal" as being in the foreground
dturner-tw
force-pushed
the
dturner/pluggable-backends
branch
4 times, most recently
from
bee0009 to
168f8ea
Compare
dturner-tw
force-pushed
the
dturner/pluggable-backends
branch
2 times, most recently
from
ed8164c to
7f6300a
Compare
dturner-tw
force-pushed
the
dturner/pluggable-backends
branch
from
7f6300a to
f796139
Compare
dturner-tw
pushed a commit
that referenced
this pull request
Sep 3, 2015
When ac49f5c (rerere "remaining", 2011-02-16) split out a new helper function check_one_conflict() out of find_conflict() function, so that the latter will use the returned value from the new helper to update the loop control variable that is an index into active_cache[], the new variable incremented the index by one too many when it found a path with only stage #1 entry at the very end of active_cache[]. This "strange" return value does not have any effect on the loop control of two callers of this function, as they all notice that active_nr+2 is larger than active_nr just like active_nr+1 is, but nevertheless it puzzles the readers when they are trying to figure out what the function is trying to do. In fact, there is no need to do an early return. The code that follows after skipping the stage #1 entry is fully prepared to handle a case where the entry is at the very end of active_cache[]. Help future readers from unnecessary confusion by dropping an early return. We skip the stage #1 entry, and if there are stage #2 and stage #3 entries for the same path, we diagnose the path as THREE_STAGED (otherwise we say PUNTED), and then we skip all entries for the same path. Signed-off-by: Junio C Hamano <gitster@pobox.com>
dturner-tw
pushed a commit
that referenced
this pull request
Sep 3, 2015
A "rebase" replays changes of the local branch on top of something else, as such they are placed in stage #3 and referred to as "theirs", while the changes in the new base, typically a foreign work, are placed in stage #2 and referred to as "ours". Clarify the "checkout --ours/--theirs". * se/doc-checkout-ours-theirs: checkout: document subtlety around --ours/--theirs
dturner-tw
pushed a commit
that referenced
this pull request
Sep 3, 2015
A "rebase" replays changes of the local branch on top of something else, as such they are placed in stage #3 and referred to as "theirs", while the changes in the new base, typically a foreign work, are placed in stage #2 and referred to as "ours". Clarify the "checkout --ours/--theirs". * se/doc-checkout-ours-theirs: checkout: document subtlety around --ours/--theirs
This is the test missing from fb86e32 (git remote: allow adding remotes agreeing with url.<...>.insteadOf, 2014-12-23): we should allow adding a remote with the URL when it agrees with the url.<...>.insteadOf setting. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: GyuYong Jung <obliviscence@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The low-level merge machinery has been taught to use CRLF line termination when inserting conflict markers to merged contents that are themselves CRLF line-terminated. * js/xmerge-marker-eol: merge-file: ensure that conflict sections match eol style merge-file: let conflict markers match end-of-line style of the context
"git worktree" had a broken code that attempted to auto-fix possible inconsistency that results from end-users moving a worktree to different places without telling Git (the original repository needs to maintain backpointers to its worktrees, but "mv" run by end-users who are not familiar with that fact will obviously not adjust them), which actually made things worse when triggered. * nd/do-not-move-worktree-manually: worktree: stop supporting moving worktrees manually worktree.c: fix indentation
"git push --force-with-lease" has been taught to report if the push needed to force (or fast-forwarded). * aw/push-force-with-lease-reporting: push: fix ref status reporting for --force-with-lease
The emulated "yes" command used in our test scripts has been tweaked not to spend too much time generating unnecessary output that is not used, to help those who test on Windows where it would not stop until it fills the pipe buffer due to lack of SIGPIPE. * js/test-lib-windows-emulated-yes: test-lib: limit the output of the yes utility
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It turns out "git clone" over rsync transport has been broken when the source repository has packed references for a long time, and nobody noticed nor complained about it. * jk/drop-rsync-transport: transport: drop support for git-over-rsync
Test scripts have been updated to remove assumptions that are not portable between Git for POSIX and Git for Windows, or to skip ones with expectations that are not satisfiable on Git for Windows. * js/mingw-tests: (21 commits) gitignore: ignore generated test-fake-ssh executable mingw: do not bother to test funny file names mingw: skip a test in t9130 that cannot pass on Windows mingw: handle the missing POSIXPERM prereq in t9124 mingw: avoid illegal filename in t9118 mingw: mark t9100's test cases with appropriate prereqs t0008: avoid absolute path mingw: work around pwd issues in the tests mingw: fix t9700's assumption about directory separators mingw: skip test in t1508 that fails due to path conversion tests: turn off git-daemon tests if FIFOs are not available mingw: disable mkfifo-based tests mingw: accomodate t0060-path-utils for MSYS2 mingw: fix t5601-clone.sh mingw: let lstat() fail with errno == ENOTDIR when appropriate mingw: try to delete target directory before renaming mingw: prepare the TMPDIR environment variable for shell scripts mingw: factor out Windows specific environment setup Git.pm: stop assuming that absolute paths start with a slash mingw: do not trust MSYS2's MinGW gettext.sh ...
An earlier adjustment of test mistakenly used write_script to prepare a file whose exact content matters for the test; reverting that part fixes the breakage for those who use SHELL_PATH that is different from /bin/sh. * mg/mingw-test-fix: t9100: fix breakage when SHELL_PATH is not /bin/sh
The automatic typo correction applied to an alias was broken
with a recent change already in 'master'.
* nd/clear-gitenv-upon-use-of-alias:
restore_env(): free the saved environment variable once we are done
git: simplify environment save/restore logic
git: protect against unbalanced calls to {save,restore}_env()
git: remove an early return from save_env_before_alias()
The "user.useConfigOnly" configuration variable can be used to force the user to always set user.email & user.name configuration variables, serving as a reminder for those who work on multiple projects and do not want to put these in their $HOME/.gitconfig. * da/user-useconfigonly: ident: add user.useConfigOnly boolean for when ident shouldn't be guessed fmt_ident: refactor strictness checks
Some calls to strcpy(3) triggers a false warning from static analysers that are less intelligent than humans, and reducing the number of these false hits helps us notice real issues. A few calls to strcpy(3) in test-path-utils that are already safe has been rewritten to avoid false wanings. * jk/test-path-utils-xsnprintf: test-path-utils: use xsnprintf in favor of strcpy
Some calls to strcpy(3) triggers a false warning from static analysers that are less intelligent than humans, and reducing the number of these false hits helps us notice real issues. A few calls to strcpy(3) in "git rerere" that are already safe has been rewritten to avoid false wanings. * jk/rerere-xsnprintf: rerere: replace strcpy with xsnprintf
The documentation for "git clean" has been corrected; it mentioned that .git/modules/* are removed by giving two "-f", which has never been the case. * mm/clean-doc-fix: Documentation/git-clean.txt: don't mention deletion of .git/modules/*
* ah/stripspace-optstring: stripspace: call U+0020 a "space" instead of a "blank"
The vimdiff backend for "git mergetool" has been tweaked to arrange and number buffers in the order that would match the expectation of majority of people who read left to right, then top down and assign buffers 1 2 3 4 "mentally" to local base remote merge windows based on that order. * dw/mergetool-vim-window-shuffle: mergetool: reorder vim/gvim buffers in three-way diffs
Add ref backend methods for: read_raw_ref, verify_refname_available, pack_refs, peel_ref, create_symref, resolve_gitlink_ref. read_raw_ref becomes static because it's not used outside refs.c Signed-off-by: Ronnie Sahlberg <sahlberg@google.com> Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a ref backend method for do_for_each_ref. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Now that we have moved do_for_each_ref into refs.c, it no longer needs to be public. Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Alternate refs backends might still use files to store per-worktree refs. So the files backend's ref-loading infrastructure should be available to those backends, just for use on per-worktree refs. Add do_for_each_per_worktree_ref, which iterates over per-worktree refs. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the file-based backend, the reflog piggybacks on the ref lock. Since other backends won't have the same sort of ref lock, ref backends must also handle reflogs. Signed-off-by: Ronnie Sahlberg <rsahlberg@google.com> Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Ronnie Sahlberg <rsahlberg@google.com> Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the file-based backend, delete_refs has some special optimization to deal with packed refs. In other backends, we might be able to make ref deletion faster by putting all deletions into a single transaction. So we need a special backend function for this. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Alternate refs backends might not need the refs/heads directory and so on, so we make ref db initialization part of the backend. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Forbid cross-backend ref renames. This would be pretty weird, but since it will break, we should prevent it. Also make the files backend deal with all non-normal ref renames. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Instead of using a files-backend-specific struct ref_lock, the generic ref_transaction struct should provide a void pointer that backends can use for their own lock data. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The check for duplicate refnames in a transaction is needed for all backends, so move it to the common code. ref_transaction_commit_fn gains a new argument, the sorted string_list of affected refnames. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The refs infrastructure learns about log-only ref updates, which only update the reflog. Later, we will use this to separate symbolic reference resolution from ref updating. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
When renaming refs, don't dereference either the origin or the destination before renaming. The origin does not need to be dereferenced because it is presently forbidden to rename symbolic refs. Not dereferencing the destination fixes a bug where renaming on top of a broken symref would use the pointed-to ref name for the moved reflog. Add a test for the reflog bug. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
When locking a symbolic ref to expire a reflog, lock the symbolic ref (using REF_NODEREF) instead of its referent. Add a test for this. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Before committing ref updates, split symbolic ref updates into two parts: an update to the underlying ref, and a log-only update to the symbolic ref. This ensures that both references are locked correctly while their reflogs are updated. It is still possible to confuse git by concurrent updates, since the splitting of symbolic refs does not happen under lock. So a symbolic ref could be replaced by a plain ref in the middle of this operation, which would lead to reflog discontinuities and missed old-ref checks. All callers to lock_ref_sha1_basic outside of ref_transaction_commit now set REF_NODEREF. And ref_transaction_commit only happens on already-derefernced refs. So we can assume REF_NODEREF when resolving inside this function. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Always handle non-normal (per-worktree or pseudo) refs in the files backend instead of alternate backends. Sometimes a ref transaction will update both a per-worktree ref and a normal ref. For instance, an ordinary commit might update refs/heads/master and HEAD (or at least HEAD's reflog). Updates to normal refs continue to go through the chosen backend. Updates to non-normal refs are moved to a separate files backend transaction. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
dturner-tw
force-pushed
the
dturner/pluggable-backends
branch
2 times, most recently
from
02e2ff1 to
7d65a82
Compare
git init learns a new argument --ref-storage. Presently, only "files" is supported, but later we will add other storage backends. When this argument is used, the repository's extensions.refStorage configuration value is set (as well as core.repositoryformatversion), and the ref storage backend's initdb function is used to set up the ref database. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
All submodules must have the same ref storage (for now). Confirm that this is so before attempting to do anything with submodule refs. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a new option, --ref-storage, to allow the ref storage backend to be set on new clones. Submodules must use the same ref storage as the parent repository, so we also pass the --ref-storage option option when cloning submodules. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
git svn learns to pass the ref-storage command-line argument (to init and clone) through to git init. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de> Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add new function register_ref_storage_backends(). This new function registers all known ref storage backends... once there are any other than the default. For now, it just registers the files backend. Call the function before calling set_ref_storage_backend. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This sets up the existing backend early, so that other code which reads refs is reading from the right place. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
It will soon be useful for resolve_ref_unsafe to support submodules. But since it is called from so many places, changing it would have been painful. So a new resolve_ref_unsafe_submodule is created instead. read_raw_ref learns a submodule argument to support this use case. The files backend doesn't need this functionality, but it doesn't hurt. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a database backend for refs using LMDB. This backend runs git for-each-ref about 30% faster than the files backend with fully-packed refs on a repo with ~120k refs. It's also about 4x faster than using fully-unpacked refs. In addition, and perhaps more importantly, it avoids case-conflict issues on OS X. LMDB has a few features that make it suitable for usage in git: 1. It is licensed under the OpenLDAP Public License, which is GPL-compatible [1]. [1] http://www.gnu.org/licenses/license-list.en.html#newOpenLDAP 2. It is relatively lightweight; it requires only one header file, and the library code takes under 64k at runtime. 3. It is well-tested: it's been used in OpenLDAP for years. 4. It's very fast. LMDB's benchmarks show that it is among the fastest key-value stores. 5. It has a relatively simple concurrency story; readers don't block writers and writers don't block readers. Ronnie Sahlberg's original version of this patchset used tdb. The major disadvantage of tdb is that tdb is hard to build on OS X. It's also not in homebrew. So lmdb seemed simpler. To test this backend's correctness, I added support to the test suite to run under LMDB. Dozens of tests use manual ref/reflog reading/writing, or create submodules without passing --ref-storage to git init. If those tests are changed to use the update-ref machinery or test-refs-lmdb-backend (or, in the case of packed-refs, corrupt refs, and dumb fetch tests, are skipped), the only remaining failing tests are the git-new-workdir tests. Later, we'll add this support to the test suite. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add tests for the database backend. Signed-off-by: David Turner <dturner@twopensource.com> Helped-by: Dennis Kaarsemaker <dennis@kaarsemaker.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a --ref-storage argument to make it easy to test alternate ref storage backends. This means that every test that calls git init or git clone must use the new ref storage argument. Modify many tests to work under alternate ref storage backends. Introduce abstractions for raw ref/reflog reading/writing in tests instead of directly frobbing the filesystem. Conditionally skip tests that are not expected to succeed under this condition. Most of this is straightforward. Of particular note are the following test changes: * The rearrangement of commands in t1401 is because without HEAD in the right place, git doesn't recognized the trash dir as a git repo, so no git commands work. * t1430-bad-ref-name specifically blocks lmdb because other alternate backends might want to keep this test. * Some tests contain comments explaining their skippage. Signed-off-by: David Turner <dturner@twopensource.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
dturner-tw
force-pushed
the
dturner/pluggable-backends
branch
from
7d65a82 to
0a3fa7f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
My, this is a lot of commits.