Community
Participate
Working Groups
This build is unusable for me. I am working on files that are stored on a network drive (ClearCase and Samba). Here is the reproducable behaviour. I open a file, modify the file then save it. Without closing the file I make another modification. When I try to save, Eclipse complains that resource out of sync. I refresh the file, Eclipse do no ask me about reloading the file, try to save again. This time Eclipse is showing me an "Update conflict" error. I have to copy the content of the file to clipboard, close the file, open the file again and paste from the clipboard. This is a bug introduced after M5. M5 has no problem. Integration build 20020510 did not shown me the "Update conflict", instead it asked if I want to reload the file after the refresh. If I said no, then press save it asked me about overwriting.
Are you running with ClearCase integrated with Eclipse, or just running separately? Are you making all the changes via Eclipse, or also modifying the files in the filesystem? Are other ClearCase users modifying the same file at the same time? Note that since M5 the "Update conflict" behaviour has changed to only warn you once and offer to reload. If you say no, it doesn't ask you again. But you should still get it.
1. I run ClearCase separately. 2. The changes are only made with Eclipse. 3. No other users are modifying the files. 4. Eclipse does not say anything about file changed. (M5 did say that). I think this behaviour is somehow linked to SAMBA. I tried to work with a project on a shared drive (Microsoft NT system) and I had no problem.
I checked with the latest build (May 20). The behaviour improved a little bit. Here is the new behaviour: 1. Open a file with Eclipse. 2. Modify the file and save it. 3. Modify the file again without reloading the file and save. Save will report resource of sync. 4. I refresh the file (by click on packages view and press F5). Now save works fine, no messages show up. I don't know how Eclipse works, but I am thinking that Eclipse mantains the modified time stamp of the file to know if somebody else changed the file or not. At the time of save however Eclipse will not read the file system again, he thinks that is has the timestamp. Is this a correct ? This problem occurs only with the ClearCase and Samba combination. I tried it on SAMBA network drive (without ClearCase) there where no problem.
Latest build ----------- The refresh, then save again works only with text files. Java files still causing problems, files can not be saved at all (it reports "Update conflict"). I have to save the content in clipboard, close the file, open it again, then paste from clipboard and save it again.
i tried the following setup: .our pessimistic temp provider (ClearCase-like) .network drive-based workspace .eclipse running on w2k and i saw no problems editing and saving files we're suspecting that that some timing problems between your samba server and your eclipse can be involved.
All testing in the editor is based on a time stamp computed with the following method of FileDocumentProvider: protected long computeModificationStamp(IResource resource) { long modificationStamp= resource.getModificationStamp(); IPath path= resource.getLocation(); if (path == null) return modificationStamp; modificationStamp= path.toFile().lastModified(); return modificationStamp; } Moving to core for consideration whether the decribed setup may cause problems in the above code.
The quoted code in FileDocumentProvider doesn't make sense. IResource.getModificationStamp() returns a value that is completely unrelated to the file's timestamp on disk. This is just a counter that we increase every time the file is somehow touched. The modification stamp can change without the local file timestamp changing, and vice-versa. However, this value is only used in FileDocumentProvider if IResource.getLocation() returns null. This almost never happens. The only case I know of is if the entire project does not exist in the workspace. The filesystem is not consulted by this method so I don't see how the Samba setup could influence this. The only remaining question for that code snippet is the granularity and accuracy of java.io.File.lastModified() when running with that Samba/ClearCase configuration. Even on a local filesystem, the granularity lastModified is in the 1-2 second range, which isn't sufficient for reliably detecting changes.
If the code does not make sense, how has the code to look like to make sense?
*** Bug 15521 has been marked as a duplicate of this bug. ***
Ideally, you should avoid all file timestamp calculations, and instead use IResource.isSynchronized() on the file. That leaves the calculation to core. We use a native method to calculate the last modified time, which may have a finer granularity than the java.io call. However, I really don't know if this code in FileDocumentProvider is the culprit in this particular case.
I tried to debug this problem. I could not. When I run it with the debugger everything is working fine. I think Clearcase updates the timestamp with a delay. If I open the same file in another editor (or Eclipse) and one of them change it, Eclipse has no problem seeing that somebody else changed it and offer me a reload. But I got an error when I try to save it more than once in the same Eclipse. I found rare cases when the out of sync error does not show up. I guess Clearcase was fast enough than.
Kai, does IResource.isSynchronized() provide you with the information that you need?
See also potentially related bug 17647.
Well bug 17647 seems to be fixed for me since F2. This "Resource is out of sync" error is a major problem for my clearcase/samba installation. One thing that would alleviate the pain would be to allow me to force an overwrite of the "out of sync" resource when I make a save.
Created attachment 1473 [details] FileSystemResourceManager.java
In my case it isn't a delay in setting the timestamp. Something is actually changing the timestamp sometime after the save by one second. I have created a workaround (see previous attachment) to do the timestamp comparison approximately rather than exactly. It produces the following log entries to demonstrate the 1 second difference !ENTRY org.eclipse.core.resources 2 0 Jun 18, 2002 13:31:34.413 !MESSAGE areNearlyTheSameLastModifiedTimes: 1024399876000, 1024399877000 !ENTRY org.eclipse.core.resources 2 0 Jun 18, 2002 13:31:49.524 !MESSAGE areNearlyTheSameLastModifiedTimes: 1024399900000, 1024399901000
Thanks for the input but I don't think that the changes give us exactly what we are looking for in the majority of cases. The FileSystemResourceManager.write method looks something like this: if the timestamp in memory is the same as on disk then write the file contents else someone has changed the file outside of Eclispe so throw an out of sync exception endif Unfortuanatly the changes which you have made to the code would do not tolerate changes made on the file system that occur within 2 seconds. This means if an external tool is making changes to the underlying files, then there is greater potential for these changes to be over-written and lost.
I agree it's not a solution. But do you appreciate how unuseable Eclipse is with this problem? Every other save fails. What about the possibility that an external application changes the file within the same second. The timestamp will not change with only 1 second accuracy. How do you want to cope with that?
Timestamp granularity is exactly one of the reasons that we use our core library to retrieve the timestamps of the file on disk. In our investigation we discovered that (depending on what VM you are running) java.io.File.lastModified() can return anywhere from milli-second accuracy to timestamp granularity of 2 seconds.
If there are problem with the java.io.File.lastModified() based on different JVM, maybe we should not use java IO but native call. Another choice could be to parametrize the granularity of the last modified time stamp. Some people system will be working correctly, other people system can work with a 2-3 seconds granularity. Is last modified time stamp is present all file systems ? M5 did not had this problem. It appeared only in the integration builds after M5. Can somebody tell us what changed ?
Well my win NT installation should be using the native library. My CoreFileSystemLibrary has not been modified from F3. As you can see in the log above the timestamp has 1 second granularity.
How about using a checksum as a secondary check when the timestamps are close but not equals?
Some more info: I'm using NT, Clearcase, Samba and Eclipse F3. Samba is configured with a timestamp granularity of 2 seconds (to support Visual Studio users). I'm not sure why we see odd timestamps at all. I think it must be something to do with the interaction of clearcase.
Just tried Rational's Clearcase plugin (2.0.18) with release 2.0. Problem still exists.
Which version(s) of Samba exhibited this behavior? I believe this problem was addressed in Samba 2.2.1. See item 17 in the "Changes in 2.2.1" section of: http://us1.samba.org/samba/whatsnew/samba-2.2.5.html
We are using 2.2.3
Summary of investigations so far: I have hacked 2.0 and now 2.01 to make the file synchronisation check treat timestamps 1 second appart as being the same. This cures this problem and that reported in bug 17647. The bug is actually in clearcase (producing inconsistent timestamps) but eclipse is rather severe in not allowing the user to overwrite the "out of synch" file. This is my current understanding of the clearcase dynamic view timestamp bug: The clearcase NT client cannot return a consistent timestamp for a given file in view storage (not vob storage). I was able to minimise the number of times this happened by ensuring my pc and the view server were timesynched but it still happens from time to time. Why the client time is involved is beyond me. Communication with another eclipse/clearcase sufferer elicited that Rational had told him that it was necessary to have Clearcase 5.00 patch 7 to have eclipse compatibility. We are on patch 3 and our admins won't upgrade at the moment. Maybe the patch fixes these timestamp problems. Can someone else confirm? So there are 2 workarounds: 1) use my hack 2) use snapshot views
*** Bug 19789 has been marked as a duplicate of this bug. ***
*** Bug 21435 has been marked as a duplicate of this bug. ***
*** Bug 17647 has been marked as a duplicate of this bug. ***
I'm attaching a simple test program that I hope will illustrate the problem. It modifies a file, then immediately asks for the timestamp. Then it delays for several seconds and asks for the timestamp again. From the symptoms we're seeing, it looks like this should give different timestamps before and after the delay. For those experiencing this problem, can you please try running the attached program from a local drive, but pointing it at a file on your samba/NFS drive? Example: C:\temp>d:jre\bin\java TimeStampTest Q:\some\path\SomeFile.txt
Created attachment 2151 [details] Test program that may illustrate the problem
Well I am not in a position to reproduce the problem at the moment because it has gone away. I am currently using a 2.1 build without my hack without problems. The clearcase admins don't think anything has changed so it must be a happy coincidence. I've even tried setting my pc clock way out but cannnot reproduce. This makes me think the problem could be caused by the clearcase servers not being synched. Nonetheless, I think I have explained what causes the problem. It is not a bug in Eclipse. It is a bug in clearcase or a misconfiguration. Eclipse, however, could be more sympathetic to the user. e.g. When compiling - write the class file anyway When saving a source file - ask the user then save it anyway BTW I only have problems accessing a clearcase dynamic view over samba, not when accessing normal nfs files over samba.
Created attachment 2153 [details] Output of the text program
I've tried the test program (output attached). There is nothing untoward there :( I still have the problem in eclipse though. Could it be the native implentation that is causing the problem? I've looked at the code a bit more, and in the class org.eclipse.core.internal.localstore.CoreFileSystemLibrary there is a getLastModified(String fileName) method which can go to a native implementation. Maybe this is the problem [wild guess]? I am happy to test it but I have no CVS access (behind firewall) so if someone wants to build a version that forces eclipse to use the non-native call to getLastModified and e-mail it to me then I will test it.
I tried the test program and here are the results (attached to a file). I tried it with Eclipse (2.0.1) and I could not reproduce the problem anymore. Clearcase and Samba is the same. I did however upgrade from Windows NT to Windows 2K
Created attachment 2154 [details] Result of test program
Well if the timestamp discrepencies are caused by server clocks being out of synch, there may be a problem with the test program waiting for an exact multiple of 1 second. Might be worth waiting for a random number of ms between each iteration and running for much longer.
As suspected, I've changed the code in the file: org.eclipse.core.internal.localstore.CoreFileSystemLibrary; as follows: public static long getLastModified(String fileName) { //if (hasNatives) // return getLastModified(getStat(fileName)); // inlined (no native) implementation return new File(fileName).lastModified(); } By commenting out the "if (hasNatives)" clause and rebuilding the jar file I have removed the bug. This, I think, conclusively shows that the problem is with the native routine. I'll attach the modified file as it make a nice workaround. But as Neil says, the actual problem is with Clearcase. The only problem with eclipse is that it doesn't prompt the user with an option to overwrite or revert the file and then put itself back in synch. I'm not entirely sure why this is, because for a "normal" concurrent modification eclipse does prompt with these options.
Created attachment 2165 [details] A modified library to use the java io for fetching last-modified times.
I'm afraid the problem does not lie with using the native library. See bug 19789. If other sufferers are in a position to, could they try to get the clearcase patches (5.0 patch 7) installed and check if that solves the problem. My admins won't do it here. John, are you in a position to discuss this with Rational. This might save a lot of time if they actually know what causes the problem.
Reproduced with modified test program. I started to get sporadic problems saving files in eclipse again so I decided to run the test program. I couldn't reproduce the problem with this test. I decided to cause the file to be read inbetween reading the first and second timestamps and then the problem was reproduceable. This is the output. I will attach the program. =============================== Problem detected after 29 iterations in 30986 ms. firstTime = 1034347853000 secondTime = 1034347852000 Problem detected after 27 iterations in 29518 ms. firstTime = 1034347882000 secondTime = 1034347881000 Problem detected after 14 iterations in 12376 ms. firstTime = 1034347894000 secondTime = 1034347893000 Problem detected after 3 iterations in 2828 ms. firstTime = 1034347897000 secondTime = 1034347896000 Problem detected after 16 iterations in 22095 ms. firstTime = 1034347919000 secondTime = 1034347918000 Problem detected after 10 iterations in 12751 ms. firstTime = 1034347931000 secondTime = 1034347930000 Problem detected after 1 iterations in 844 ms. firstTime = 1034347933000 secondTime = 1034347932000 Problem detected after 126 iterations in 138305 ms. firstTime = 1034348072000 secondTime = 1034348071000 Problem detected after 55 iterations in 66411 ms. firstTime = 1034348137000 secondTime = 1034348136000 Problem detected after 24 iterations in 26783 ms. firstTime = 1034348165000 secondTime = 1034348164000 =============================== Note that we are currently using dos timestamps in samba with a 2 second granularity to support visual studio users. I have experienced the saving problem in eclipse even with dos timestamps switched off though.
Created attachment 2169 [details] This test program will reproduce the problem
I was able to reproduce the problem with the new test code. Here is the result: ================================================================== Problem detected after 423 iterations in 451188 ms. firstTime = 1034350862000 secondTime = 1034350863000 Problem detected after 95 iterations in 102171 ms. firstTime = 1034350963000 secondTime = 1034350964000 Problem detected after 142 iterations in 153078 ms. firstTime = 1034351117000 secondTime = 1034351118000 Problem detected after 71 iterations in 72344 ms. firstTime = 1034351190000 secondTime = 1034351191000 Problem detected after 21 iterations in 23094 ms. firstTime = 1034351212000 secondTime = 1034351213000 Problem detected after 171 iterations in 182562 ms. firstTime = 1034351395000 secondTime = 1034351396000 Problem detected after 295 iterations in 315688 ms. firstTime = 1034351710000 secondTime = 1034351711000 Problem detected after 31 iterations in 27140 ms. firstTime = 1034351737000 secondTime = 1034351738000 Problem detected after 93 iterations in 96157 ms. firstTime = 1034351834000 secondTime = 1034351835000 Problem detected after 89 iterations in 104000 ms. firstTime = 1034351937000 secondTime = 1034351938000
I ran the modified test program for 5 hours without any problems. I'm speculating that perhaps the native routines are faster (in some sense) than the java routines and so cause me problems. My samba connection might be sufficiently slow that I don't get any problems when using the java IO. Does this sound plausable?
Can you find out what patch level of clearcase you are running. According to http://www.rational.com/support/downloadcenter/addins/clearcase/plugins/readme_C learCase_w.html You should be at patch 7 for CC 5.0
We are on clearcase 4 still :( I asked our admins, they plan to upgrade to 5 at some point at which point they can look at applying patches. Until then I'm stuck with the old version.
There is also a patch for 4.2 that fixes the same bug. http://www.rational.com/techsupport/clearcase/tech/patchlink/new/htmlstreams/clearcase_p4.2.NT.html#clearcase_p4.2.NT-17
I found the following relevent clearcase bugs: =============== Corrects a defect that could cause the modified time of a file in a dynamic view with storage on a non-FAT file system to be set as much as two seconds ahead of the current system time. http://www.rational.com/techsupport/clearcase/tech/patchlink/new/htmlbugs/CMBU00059904.html =============== Corrects a defect in the integration between ClearCase and IBM WebSphere Studio Application Developer (WSS AD). http://www.rational.com/techsupport/clearcase/tech/patchlink/new/htmlbugs/CMBU00060711.html
Have applied Clearcase 5.0 NT patches 8 and 12. My test program still reproduces the problems :-(
We suffer from this problem, too. 5 Persons work with Eclipse/Clearcase, 3 never get the problem and 2 nearly always. If we compare PC NT system times, they differ exactly this one second which also CoreFileSystemLibrary.getLastModified(stat) and info.getLocalSyncInfo() in FileSystemResourceManager.write() differ during our tests. We still use Clearcase 4.2 and will not upgrade soon. Before having found this bug report, we patched FileSystemResourceManager.write(), setting force=true, but testing for a 2 second difference is the better workaround, we'll change that.
Created attachment 3389 [details] Clearcase over samba workaround My earlier workaround/hack applied to the M5 build
Created attachment 3425 [details] improved Clearcase over samba workaround Found another place where timestamps are compared that occurred during refactoring.
I can keep updating my patch for the rc builds if anybody is interested.
I'm interested although I must admit the last one didn't work for me :( I would like to investigate furthur but I don't really have the time. If only there was an "Overwrite Anyway" option on the error dialog. Then I could live with it!
Created attachment 3976 [details] workaround patch for 2.1rc2
Created attachment 4393 [details] patch for 2.1 full release
Created attachment 5250 [details] patch for 3.0M1
I finally got round to raising an issue with Rational. This was the reply: "this is a known defect, RATLC00693656" I can't find a way to view defects online though. It would be nice to know whether XDE or WSAD suffer the same problems since these will presumably be frequently used with Clearcase.
The full details of my enquiry to Rational support follows: From: IBM Rational Software Customer Services (Europe) Hi Neil, actually, the development think it is. The thing is that is due to the MVFS and cannot be fixed until schema 53 has been retired. At the moment I cannot tell you when this will be. -----Original Message----- Hi Mikael, I don't think this is the same problem. The java program I wrote to reproduce the bug does the following. 1. create a file 2. get it's last modified time 3. wait for a random time up to 2 seconds 4. read the file 5. get it's last modified time (should be the same as from 2) This is repeated until the 2 last modified times are different. This generally happens after about 10-15 iterations. The times always differ by 1 second, 2 when we use Samba Dos timestamp resolution. The fact that the times are only out by 1 second makes me think it has nothing to do with the defect you told me about. -----Original Message----- From: IBM Rational Software Customer Services (Europe) Hi Neil, no, you cannot. All defects are logged in an internal database, and you will only see those defects that has been fixed in a patch. I cannot give you all the content in the defect, but here is the description for it: The creation time on files under version control displayed from windows explorer is not correct. If an element is checked out and you view the properties of the file, you see that the creation time of it is exactly the same as the last access time. Even worse, the access time is actually the time when the properties windows gets open, not when the file was read last time. The Windows Explorer gets the wrong information from the mvfs. This is not a problem from a pure windows explorer perspective since the version tree browser provides the correct information on version creation dates etc. But it is a problem when ClearCase interacts with other tools. Emacs for example. From time to time Emacs wants to reload the file because it runs a stat on it and the information it receives is the same as the one windows explorer displays. -----Original Message----- Mikael, can I view details of this defect on the www. I can't find it on rational.net. -----Original Message----- From: IBM Rational Software Customer Services (Europe) Dear Neil Swingler, regarding your service request 190222159; this is a known defect, RATLC00693656, and I have added you to the customer list. You will be notified when this has been fixed. -----Original Message----- There has been a longstanding bug against eclipse: https://bugs.eclipse.org/bugs/show_bug.cgi?id=16280 Our environment is win NT client and solaris server using MVFS/samba and dynamic views I investigated this and found that MVFS doesn't return a consistent timestamp for files on the view server so this is a clearcase bug, not an eclipse bug. A program is attached to the bug that can demonstrate the problem. Eclipse queries the file to get it's timestamp. Then it reads the file and queries the timestamp again. ~5% of the time, the 2 timestamps will differ by 1 second (2 seconds if Samba is configured to Dos timestamp resolution). Eclipse claims the files are out of sync and refuses to carry on doing whatever it is doing. Client and server are Clearcase 5.0. Client is at fix pack 15.
I have created a new patch for 3.0m3: http://uk.briefcase.yahoo.com/neil_swingler and go to the eclipse folder In 3.0m3 there is yet another place where the timestamp check is made that I had to change.
The yahoo briefcase doesn't have public folders anymore so I've put the patch at http://www.neil.swingler.name/eclipse/index.html instead
I am going to close this bug report, since it has been raised as a bug against Rational and they have acknowledged the bug. This is not an Eclipse defect. Neil, you are welcome to continue posting your workaround here for those who are interested - thank you for providing this. I hope you understand that this is not an acceptable workaround for us to release into the core, but I'm glad that it's alleviating the pain a bit.
I think you should be pragmatic and take into account the clearcase bug. You are all IBM (for the moment). It is going to be a while before the next Clearcase schema becomes available and could be many more years before everybody is using it. Most users who pay lots of money for CC and WSAD will never read this bug report or find out about my patch. They will just get completely frustrated.
Patch for 3.0m4: http://www.neil.swingler.name/eclipse/index.html
Patch for 3.0m5: http://www.neil.swingler.name/eclipse/index.html Be patient, the connection is only 64 kbit/sec
Sorry previous URL won't work. Try this one: http://www.neil.swingler.name/eclipse
3.0m6 version patch at: http://www.neil.swingler.name/eclipse
*** Bug 28432 has been marked as a duplicate of this bug. ***
3.0m7 version patch at: http://www.neil.swingler.name/eclipse
Patch updated for 3.0 release http://www.neil.swingler.name/eclipse I will continue to update the patch for milestone builds but won't polute this bug report with a comment each time.
It turns out that this is *not* a bug in ClearCase. It's a Samba bug. Here are the details from the engineer that tracked this down: When smbd receives SMB Transact2 SetFileInfo command setting the modification timestamp of the opened file, it keeps the timestamp in the pending_modtime field and skips utime(). (See call_trans2setfilepathinfo() in "source/smbd/trans2.c".) Thus, the follow-on SMB Transact2 QueryFileInfo command asking the modtime of this opened file returns the stale information (despite the fact that SetFileInfo command succeeded). The modtime is updated when SMB CloseFile command is received. If Eclipse remembered the stale modtime (returned before SMB CloseFile, but after SMB Transact2 SetFileInfo), the next open will get the resource out of sync error (because the modtime has been changed). Obviously, the simple fix in smbd is to call file_utime() in call_trans2setfilepathinfo() right after setting pending_modtime.
Thanks for the feedback. Do you have a samba bug ID for that? Samba bugzilla search sucks even more than eclipse bugzilla :-)
IBM raised this Samba bug: https://bugzilla.samba.org/show_bug.cgi?id=1729
Patch updated for 3.0.1 and 3.1m2: http://neil.swingler.name/eclipse
The samba bug 1729 https://bugzilla.samba.org/show_bug.cgi?id=1729 has been fixed. It will be in Samba 3.0.10 !
I have patched m4. Hopefully for the last time. Samba 3.0.10 is out but we can't try it out right now. If anybody can try Samba 3.0.10, please report here whether the problem is solved.
The fix for this bug appears to be in samba 3.0.11-pre1: http://websvn.samba.org/cgi-bin/viewcvs.cgi/branches/SAMBA_3_0_RELEASE/source/smbd/trans2.c?rev=4348&view=markup I haven't tested the official build yet (will test when 3.0.11 hits debian testing) yet, but I have tested samba with the patch attached to the samba bug for this problem, and it appears to make it much bette_, although I still get a few (every 10-20 saves) 'resource out-of-sync' errors, perhaps due to differing clocks on the client & server computer? I am not sure.
neil.swingler.name is now dead The patch (now available for 3.1) is available at http://swingler.ch/eclipse
Created attachment 36486 [details] patch for 3.2 m5 Since my website is in a bad state I will continue posting the patch here
I just filed a similar bug which does still occur with samba 3.0.22 using cifs: https://bugs.eclipse.org/bugs/show_bug.cgi?id=137011
