Issue 695
| Description: |
Opened: Wed May 1 12:40:00 -0700 2002 |
Sort by: Oldest first | Newest first |
Suppose you check out the top-level of the Subversion
repository with --nonrecursive, so you just get the
README, INSTALL, HACKING files etc, but none of the subdirs.
Now you have a flat, one-level working copy with just some
files.
Currently, if you run "svn update" in that working copy,
everything goes to pot. The update state report is incorrect:
the client doesn't tell the server anything about subdirs
(because the client has no record of any subdirs, neither on
disk nor in the entries file), so the server assumes that the
client has revision X of the entire tree, where X is whatever
the client got when it checked out.
Depending on what has changed in the repository since X, various
sorts of errors will happen at this point. No need to go into
it, you can use your imagination.
So what is the desired behavior?
Nonrecursiveness should be sticky. That is, when you check out
something --nonrecursive, the local wc directory created should
remember that it is a nonrecursive tree, and tell that to the
server in state reports, so the server doesn't try to send
updates for anything not that directory level.
How would the working copy record this? Probably by setting
an attribute on the `this-dir' entry, plus recording all the
subdir entries with a similar attribute on them.
So "svn update" in this working copy would behave non-recursively
by default; but if you pass the --recursive flag, then it
clears those attributes and fetches everything. That way, the
intention expressed in the initial checkout is honored, until
the user says otherwise.
This also implies that the --recursive vs --nonrecursive state
variable in the client needs to become a three-state enum:
--default, --recursive, and --nonrecursive.
------- Additional comments from Karl Fogel Wed May 1 12:52:36 -0700 2002 -------
Marking as bite-sized.
This is entirely a client-side issue. The working copy needs
to record various things, and the RA layer needs to toggle
the `recurse' flag passed to do_update() and/or use the RA
reporter's delete_path() to report missing paths. The combination
of those two abilities allows a nonrecursive working copy to
a) Stay nonrecursive (which is what it should be by default)
or
b) Fill out recursively (if the users explicitly asks)
Volunteers?
------- Additional comments from Greg Stein Tue Jun 11 16:41:18 -0700 2002 -------
Removing the "bite-sized" milestone, and switching to a keyword. Set the
appropriate milestone (Beta) based on this issue's priority.
------- Additional comments from C. Michael Pilato Wed Jul 31 12:48:02 -0700 2002 -------
p5, post-1.0
------- Additional comments from Robert Schiele Mon Sep 23 11:28:14 -0700 2002 -------
As suggested by Karl Fogel I will describe here an addition to the
co command (Mostly a copy from my mail to the dev list --- for the
full discussion see the mailing list archive.):
Currently svn co needs at least one URL.
A new operating mode should be introduced for co which would be
used, if no URL bot only a relative directory is given. Valid
options are the same as for "normal" co mode. This means
# svn co subdir
would do the following:
- If called outside a working copy, print an error message.
- If called inside a working copy of the URL
http://server/path/to/dir it will:
- Print an error message, if subdir already exists as file or dir
in the current directory.
- Add http://server/path/to/dir/subdir to the current working
copy, if the repository contains such a subdir and there is no
conflicting file or dir in the current working copy.
- Print an error message, if http://server/path/to/dir/subdir does
not exist in the repository.
The operation that should be done in the non error case should be
similar as:
# svn co http://server/path/to/dir/subdir
Add '<entry name="subdir" kind="dir"/>' to .svn/entries in current
dir.
------- Additional comments from Kevin Pilch-Bisson Mon Jan 27 09:59:12 -0700 2003 -------
Assigning to self
------- Additional comments from Philip Martin Sun Mar 14 11:32:19 -0700 2004 -------
*** Issue 1785 has been marked as a duplicate of this issue. ***
------- Additional comments from Philip Martin Wed Mar 31 05:34:33 -0700 2004 -------
As currently implemented, non-recursive checkout is fragile and breaks easily.
It occurs to me that it might work better if the non-recursive versions of
checkout, update and switch were to leave the working copy marked incomplete.
This is not the same as making non-recursive sticky, but it's probably much
easier to implement and it could well make things less fragile.
------- Additional comments from Karl Fogel Mon Apr 5 06:39:35 -0700 2004 -------
(Noting that possibly issue #1634 and issue #1768 are of interest, although
closed now.)
------- Additional comments from Ben Collins-Sussman Mon May 24 08:58:43 -0700 2004 -------
I'm upping the priority of this issue, because apparently 3.9e57 former CVS
users are outraged. The ability to 'svn co -N' and then selectively make
subdirectories "appear" is a really, really common CVS ability that users have
come to rely on. (Though apparently no svn developer has ever used or cared
much about this feature.)
The current situation is bleak: as kfogel originally points out at the top of
this issue, 'svn co -N' creates a broken working copy. Really, I mean *broken*.
The entries file has *no* entries representing subdirectories, and yet there's
absolutely nothing in the file to indicate incompleteness. The entries file
thinks it's complete! I mean, it's one thing for an update to "lie" to the
server about what the working copy does and doesn't have; but in this case, the
working copy doesn't even *realize* it's incomplete. Yikes.
I'm wondering if anyone has the time/energy to work on this for svn 1.1. It
involves a bunch of libsvn_wc work: not just teaching the entries file to
understand that certain subdirs are "unexpanded", but giving users the ability
to "expand" them somehow. Maybe philip or jpieper would be interested.
------- Additional comments from Greg Hudson Mon May 24 10:38:33 -0700 2004 -------
In the interest of having a workaround, even if it isn't a good one:
It's been pointed out that one way to get a non-broken, "non-recursive" working
copy is to "svn up -rNNNN dir", where NNNN is the revision where dir was created
(hopefully with not much in the way of contents, although that won't be true if
the directory was created with a big import).
From there, you could conceivably cd into the directory and use "svn up" to
create a variant-revision working copy where some but not all of the subdirs are
present. The problem: "svn up wcpath" refuses to operate on a wcpath which is
not currently under version control, even if it exists in the target rev. But
it would probably be easier to remove that limitation than it would be to fix
non-recursive checkouts.
This isn't a very good workaround even if it's made to work, because you have to
find the rev when the directory was created, and hope that it was created
without much contents. But it's a start.
------- Additional comments from Ben Collins-Sussman Fri Jun 4 14:24:29 -0700 2004 -------
By the way, here's another variant of this bug:
$ cd working-copy
$ rm -rf subdir
$ svn up -N
D subdir # huh?
$ svn up
# nothing happens, dir never comes back.
Go look at the entries file, and notice that 'subdir' is completely gone, and
the entries file isn't marked incomplete. The entries file is broken and not
even aware of that fact. Ugh.
------- Additional comments from Branko Cibej Mon Jun 7 21:03:12 -0700 2004 -------
Might I suggest that we disable the -N option for "co" and "up" in 1.1, unless
we can fix this by then? It's better to get a "sorry not implemented" than to
silently break the working copy.
------- Additional comments from Ben Collins-Sussman Tue Jun 8 19:15:48 -0700 2004 -------
Assigning to 1.1-consider as part of the 1.1 issue-sweep.
Branko's suggestion of disabling 'co/up -N' seems good to me. If we don't have
serious time to really fix the problem (it will likely be a lot of work), then
it's better to disable this feature altogether in 1.1. Not only does the -N
flag mislead CVS users (they expect to be able to make subdirs appear at will,
as they do in CVS), but it actually creates *broken* entries files.
------- Additional comments from Ben Collins-Sussman Mon Jun 28 17:47:33 -0700 2004 -------
On second thought, ghudson points out that a lot of users may very well be using
'svn co -N' and living happily with it. Not all of them are necessarily trying
to make subdirs reappear; they may just be blissfully ignorant of the fact that
they have broken entries files. If we disable the flag altogether, we may end
up with hordes of angry users at our door, brandishing torches and pitchforks.
So I'm putting this in svn 2.0. A bunch of us in IRC today believe that we'll
need "weeks" to come up with the Correct Design for this feature.
------- Additional comments from Michael W Thelen Mon Jun 28 22:25:31 -0700 2004 -------
Since that's the case, I'm removing the "bite-sized" keyword... It seems that
this issue's size will be at least a few mouthfuls. :-)
------- Additional comments from Josh Pieper Tue Jul 6 13:36:14 -0700 2004 -------
r10153 masks some of the seriousness of this bug when you are trying to make
directories re-appear in a non-recursive checkout. After the change, you
actually can make directories re-appear, but the entries file is still broken.
------- Additional comments from Ben Collins-Sussman Thu Aug 12 08:09:31 -0700 2004 -------
*** Issue 1978 has been marked as a duplicate of this issue. ***
------- Additional comments from Geoff Rowell Tue May 3 05:53:49 -0700 2005 -------
This issue also has implications for the handling of externals. If the "--non-
recursive" flag is set, the externals of the folder should not be retrieved.
------- Additional comments from Kevin Pilch-Bisson Tue May 3 09:18:58 -0700 2005 -------
I'm not likely to work on this anytime soon.
------- Additional comments from Max Bowsher Tue May 3 10:02:22 -0700 2005 -------
'issues' is a tigris-wide mailing list.
It should be 'issues@subversion'.
------- Additional comments from Ben Reser Sun Jul 10 15:38:31 -0700 2005 -------
Delete 2122 from this issue temporarily.
------- Additional comments from Ben Reser Sun Jul 10 15:40:58 -0700 2005 -------
reassign to myself
------- Additional comments from berto Fri Aug 19 23:40:28 -0700 2005 -------
I read through the proposal for fixing this issue and the solution seems more
difficult than necessary. Because it requires you to learn two two more
commands, it also makes SVN harder to use. The nice thing about CVS's behavior
is that it doesn't really care if the directory is there or not. If a versioned
directory is checked out, it checks for modifications, otherwise it's just as
happy as if the directory wasn't there. The same simplicity should be available
in SVN.
I've run some quick tests with a very very basic tree structure and
it looks like my proposed solution below would work using the functionality
that's already available. Here's what I suggest:
A non-recursive checkout would request a complete .svn directory, but not copy
any other files. As a test to this mechanism, I created the following structure
in a repository:
a
b
b/ba
b/bb
b/bc
c
checked it out, and then blew away all the files (rm -rf *) from the working
copy root, minus the .svn directory. Now, when I check the status of that
directory:
[berto@pbnj][585]$ svn st
! a
! b
! c
it tells me all these directories are missing; the entries file knows about the
directories, but they are flagged as not being there. Ideally, this could be
quieted down, possibly by adding a flag (e.g. -qm for quiet missing), so it
won't notify these 'missing' entries.
So, now, if I want to fill in part of my working copy, I simply svn up
the particular directory I need:
[berto@pbnj][587]$ svn up b
A b
A b/ba
A b/bb
A b/bc
But, suppose I just wanted b/ba instead of all of the items underneath b. At
this point it'd be nice if the command:
svn up b/ba
was smart enough to see that b wasn't in the working copy, update it
non-recursively and then run an update on b/ba to 'fill' my working
copy. Instead, it errors out when it can't find the directory b:
[berto@pbnj][521]$ svn up b/ba
svn: 'b' is not a working copy
In order to simulate this behavior, I ran the following:
[berto@pbnj][588]$ svn up b && rm -rf b/* && svn up b/ba
and the status of the working copy root:
[berto@pbnj][593]$ svn st
! a
! b/ba
! b/bc
! c
This seems like a very simple way to get non-recursive behavior and
not have to add any additional commands. The modifications needed
would be:
- svn {co,up} -N would checkout a full .svn directory, yet not download any
files; i.e. this should work:
svn co -N mech://path/to/svn # downloads just the .svn dir
svn up b/bc # only fills in b/bc
I should end up with only b/bc in my working copy
- make svn st more quiet when it comes to reporting missing directories
Thanks!
------- Additional comments from Branko Cibej Sat Aug 20 01:46:15 -0700 2005 -------
berti saith:
> I read through the proposal for fixing this issue and the
> solution seems more difficult than necessary. Because it
> requires you to learn two two more commands, it also makes
> SVN harder to use. The nice thing about CVS's behavior is
> that it doesn't really care if the directory is there or not.
You're forgetting a fundamental difference between CVS and SVN: SVN versions
directories, CVS does not.
> - svn {co,up} -N would checkout a full .svn directory, yet not download
> download files;
This seems totally wrong behaviour; it's not a non-recursive checkout.
> i.e. this should work:
>
> svn co -N mech://path/to/svn # downloads just the .svn dir
> svn up b/bc # only fills in b/bc
>
> I should end up with only b/bc in my working copy
That works now, yes, but if you do an ordinary "svn up" in the root of your WC,
you'll get everything. You'd have to always update with explicit file and
directory names -- that's not making SVN easier to use.
> - make svn st more quiet when it comes to reporting missing directories
Excuse me? How? By not telling you about missing directories at all? That's
contrary to the whole idea of versioned diretories.
------- Additional comments from berto Mon Aug 22 14:34:14 -0700 2005 -------
> > - svn {co,up} -N would checkout a full .svn directory, yet not download
> > download files;
>
> This seems totally wrong behaviour; it's not a non-recursive checkout.
Then maybe call it a blank checkout; something like "svn --blank"? Basically a
way to link a working copy to some path in a repository, but leave it completely
empty. This sounds useful in case you don't want any files that may be
sprinkled within the directory being checked out either.
> That works now, yes, but if you do an ordinary "svn up" in the root of your
> WC, you'll get everything. You'd have to always update with explicit file and
> directory names -- that's not making SVN easier to use.
Right you are; sorry. The commands svn [include|exclude] make sense; if a
file/directory is not included, svn up would not try to update it.
> > - make svn st more quiet when it comes to reporting missing directories
>
> Excuse me? How? By not telling you about missing directories at all? That's
> contrary to the whole idea of versioned diretories.
Just to clarify, in my initial post I mention quieting down the reporting by
specifying a -qm (quiet missing) flag. Not displaying missing directories would
be an explicit request.
If something is not included using "svn include", would "svn st" display a line
saying those entries are exluded? A "-qe" flag (quiet excluded) still sounds
like a good idea. Or maybe in this case, it'd be a verbose flag, so it doesn't
show the exluded entries by default.
------- Additional comments from Karl Fogel Thu Jul 13 11:08:05 -0700 2006 -------
Starting.
------- Additional comments from Daniel Rall Thu Apr 12 09:04:20 -0700 2007 -------
Karl, won't this issue be resolved in 1.5?
------- Additional comments from Karl Fogel Thu Apr 12 12:43:36 -0700 2007 -------
Thanks for noticing the mis-milestoning of this.
I've put it in '1.5' -- unbreaking 'checkout -N' is a subset of sparse-directories.
------- Additional comments from Daniel Rall Mon Apr 16 10:08:44 -0700 2007 -------
The initially reported use case is currently working on trunk (against a 1.4.3
server, no less):
$ svn --version
svn, version 1.5.0 (dev build)
compiled Apr 4 2007, 19:59:05
...
$ svn co --depth=files http://svn.collab.net/repos/svn/tr
unk /tmp/trunk
$ svn co --depth=files http://svn.collab.net/repos/svn/tr
unk /tmp/trunk
A /tmp/trunk/Makefile.in
A /tmp/trunk/build.conf
A /tmp/trunk/win-tests.py
A /tmp/trunk/configure.in
A /tmp/trunk/TRANSLATING
A /tmp/trunk/COMMITTERS
A /tmp/trunk/README
A /tmp/trunk/BUGS
A /tmp/trunk/HACKING
A /tmp/trunk/INSTALL
A /tmp/trunk/COPYING
A /tmp/trunk/CHANGES
A /tmp/trunk/autogen.sh
A /tmp/trunk/gen-make.py
A /tmp/trunk/dist.sh
A /tmp/trunk/aclocal.m4
U /tmp/trunk
Checked out revision 24587.
$ cd /tmp/trunk/
$ svn up
At revision 24587.
$ ls
aclocal.m4 build.conf configure.in gen-make.py* Makefile.in win-tests.py
autogen.sh* CHANGES COPYING HACKING README
BUGS COMMITTERS dist.sh* INSTALL TRANSLATING
------- Additional comments from Mark Phippard Tue Jul 17 18:18:25 -0700 2007 -------
This almost could be closed. With current trunk @ r25777, this is what I get:
$ svn co -N http://svn.collab.net/repos/svn/trunk non-recursive
A non-recursive/Makefile.in
A non-recursive/build.conf
A non-recursive/win-tests.py
A non-recursive/configure.in
A non-recursive/TRANSLATING
A non-recursive/COMMITTERS
A non-recursive/README
A non-recursive/BUGS
A non-recursive/HACKING
A non-recursive/INSTALL
A non-recursive/COPYING
A non-recursive/CHANGES
A non-recursive/autogen.sh
A non-recursive/gen-make.py
A non-recursive/dist.sh
A non-recursive/aclocal.m4
U non-recursive
Checked out revision 25777.
$ svn up non-recursive/
At revision 25777.
At this point it seems like it is all working. Just for the heck of it, I wanted to show that this just now always does the right thing.
But look what happened:
$ svn up -r 25700 non-recursive/
U non-recursive/build.conf
svn: REPORT request failed on '/repos/svn/!svn/vcc/default'
svn: Directory 'non-recursive/www' is missing
svn: Directory 'non-recursive/www' is missing
Even worse, an svn up now brings down everything. I imagine I could have said --depth=files when I specified the revision, but I see no reason I should have to.
------- Additional comments from Domen S. Sun Aug 19 23:18:17 -0700 2007 -------
As it was previousely stated, this method should work also:
svn co http://server/rep/trunk -N
avn add http://server/rep/trunk/sub1/sub2/sub3
If sub1 and sub2 and sub3 exists this folders should be checked-out (only the
stated folders with their contents, not other subfolders in them) and the path
should be added to the working copy to link it with it (.svn/entries).
If let's say sub2 and sub3 don't exist in repository, they should be added with
"add" command.
------- Additional comments from Karl Fogel Mon Aug 20 11:45:03 -0700 2007 -------
We shouldn't conflate adds and checkouts.
If you're adding new directories, they are local paths, not URLs.
If you're pulling into the working copy a subdirectory tree (even a sparse one)
that already exists in the repository, then the "add" command shouldn't do it,
but rather "checkout" and possibly "update".
------- Additional comments from Mark Phippard Mon Sep 10 12:52:19 -0700 2007 -------
With trunk @ r26511 this problem still happen when accessing a 1.4 server:
U non-recursive/build.conf
U non-recursive/win-tests.py
subversion/libsvn_ra_neon/util.c:1279: (apr_err=155005)
svn: REPORT request failed on '/repos/svn/!svn/vcc/default'
subversion/libsvn_wc/lock.c:988: (apr_err=155005)
svn: Directory 'non-recursive/www' is missing
subversion/libsvn_wc/lock.c:988: (apr_err=155010)
svn: Directory 'non-recursive/www' is missing
The same test with a 1.5 server works OK
------- Additional comments from Karl Fogel Thu Sep 20 14:32:30 -0700 2007 -------
This IRC chat from today tells our diagnosis and thoughts on fixing:
<kfogel> cmpilato: figured out the cause of the problem on dev@
whereby 'svn co -rOLD_REV --depth=empty
http://svn.collab.net/repos/svn/ wc; svn up wc' results in
an error.
<cmpilato> rounder every day.
<kfogel> Basically, we add on the depth-filtering editor based on
the *requested* depth of the operation.
<cmpilato> yessir.
<kfogel> But in the latter 'svn up', the requested depth is unknown
(hence infinity). The wc itself has depth-empty, of
course.
<kfogel> So no filtering happens.
<kfogel> Hmmm.
<cmpilato> in which case, the depth filter does nothing.
<cmpilato> right.
<kfogel> Wait, what's supposed to happen to that wc?
<kfogel> Update to infinity?
* kfogel doesn't even know
* kfogel consults sparse-directories.txt
<kfogel> or stay at depth=empty?
<cmpilato> i thought it was supposed to stay at depth=empty.
<kfogel> I think so too.
<cmpilato> that 'svn up' meant "preserve my previous depth selections"
<kfogel> yes, that's right
<kfogel> Just making sure -- yes, all agree on this.
<kfogel> Pondering how to make the filtering editor aware of the
local depths, which may be mixed...
<cmpilato> ewww.
<cmpilato> the filtering editor has a well-defined job, and it does
it.
<kfogel> should this not be part of that job?
<cmpilato> i'm not sure. he's a salaried worker, so i guess we can
abuse him as needed.
<kfogel> I thought he was an intern?
* kfogel makes an obscure lisp joke
* cmpilato notes the redundancy of "obscure" in that statement.
<kfogel> cmpilato: I think we might need to add the filtering editor
unconditionally (or rather, only on the condition that the
server is not depth-aware), and then as it gets called, it
inspects the local depths and determines what to pass
through.
<kfogel> Does this sound sane?
<kfogel> In other words, both revert r26709, and make some more
enhancements to the filtering editor.
<kfogel> I'm not proposing you do it, just checking with you before
I do it.
<cmpilato> kfogel: honestly, i'm unable to think clearly right now.
my environment is ... unsupportive of attempts to do so.
<kfogel> cmpilato: no problem
<cmpilato> i've been wanting to phone-chat with you about exactly this
scenario, though.
<cmpilato> are you able to defer action until tomorrow?
<kfogel> Yes, I can work on other things.
<kfogel> In the meantime, I'll file an issue about this.
<kfogel> Or rather, annotate issue #695.
<cmpilato> cool.
<cmpilato> what time are you able to chat in the morning?
<kfogel> oooooh
<kfogel> tomorrow sucks
<kfogel> 9am-4pm booked
<cmpilato> what about tonight?
<kfogel> tonight from about 5pm Pacific also booked
<cmpilato> (i'll be away until probably 9pm your time)
<kfogel> weekend good
<kfogel> understand if you don't want to do work stuff on weekend,
though
<kfogel> monday good
<cmpilato> tentative weekend call, then. monday fallback.
<kfogel> ok
Further notes: this may imply that RA context objects need to keep requested
depth of svn_depth_unknown distinct from svn_depth_infinity for longer. For
example, this line in subversion/libsvn_ra_neon/fetch.c:make_reporter():
rb->depth = (depth == svn_depth_unknown ? svn_depth_infinity : depth);
may need to preserve the actual depth, so that later when the filtering editor
is added, it can know the actual depth requested and DTRT with incoming editor
calls.
------- Additional comments from Karl Fogel Mon Sep 24 00:06:35 -0700 2007 -------
Okay, I went down that road a bit, and then pulled back in a fit of sanity.
Here's the thing:
In order for the depth-filtering editor to know what the local depth should be
at or under a given directory in the edit, it would need to read the depth from
.svn/entries. In other words, something like this: okay_to_edit() in
libsvn_delta/depth_filter_editor.c would probably have to take a 'const char
*path' parameter, and change its return type to 'svn_error_t *' (setting *okay
by reference). It would read in the path's depth, calculate whether path should
be affected by this drive, and set *okay according to that.
But let's step back for a second: this is plainly ridiculous. The editor in
libsvn_wc/update_editor.c *already* reads depths from the working copy. It
should just do the depth-filtering itself, instead of us doubling the number of
system calls to achieve the same effect via a separate editor.
It may be that the depth-filtering editor should continue to filter out based on
requested depth, while the regular update editor filters out based on local
depth (the union of these two filterings is the end goal). Or it may be that it
should all happen in the regular update editor: that is,
svn_wc_get_update_editor3() takes a depth parameter. Oh look, it already does,
hmm. We just need to teach it to DTRT with depth==unknown.
Thinking out loud here: a possible implementation strategy is to keep the
depth-filtering editor's code separate, but move it into update_editor.c and
invoke it from within the corresponding functions of the regular update editor.
That might actually be the most maintainable way. The best thing is, then we
could just share baton knowledge, and use the batons to pass the depth
information from the regular update editor (which has the information) to the
depth-filtering editor (which needs it).
Anyway, I'm reverting my changes to depth_filter_editor.c. These changes need
to go in update_editor.c.
------- Additional comments from Karl Fogel Mon Sep 24 18:24:57 -0700 2007 -------
Tracking further progress at issue #2842.
------- Additional comments from David Glasser Mon Oct 15 17:12:29 -0700 2007 -------
Vlad fixed two of the failures with r27195. Test 19 still fails (it is trying
to find text-base for nonexistence A/mu during the first diff call)... because
we need to add ambient depth to *all* of our client side editors, like the diff
editor!
Sigh.
------- Additional comments from Karl Fogel Mon Oct 15 22:45:33 -0700 2007 -------
Well, that should be doable! I don't think I can get to it during this week's
conference, but I'll be back on the weekend.
------- Additional comments from Karl Fogel Wed Jan 9 10:10:46 -0700 2008 -------
AFAIK this is done (see dependency issues). Closing.
