 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Windows Server Forum / Windows 2000 / File System / June 2008Since weekend update, small files up to 4096 bytes get corrupted
Hi, I have two different servers (different models and manufacturing years), but both of them running Windows 2000 server with software mirrored NTFS drives, which suddenly corrupt small image thumbnail files after an hour or two. No larger file sizes seem to be effected. The file content is set to all binary 0xdf. If we replace those files with valid copies, then those files will appear to be correct for a while (probably read back from cache) but after some time they will "revert" back to all 0xdf. I have since been in contact with another customer who has the exact same problem with his Windows 2000 server also since around the past weekend. In the days prior we did install recent security fixes and Microsoft product updates that were released in the past month. Is there anyone else out there with the same problem? Multi-posted - see your other posts. Please use cross-posting if you wish to address several newsgroups. Multiposting wastes everybody's time. > Hi, > [quoted text clipped - 15 lines] > > Is there anyone else out there with the same problem? YES I have been in a losing battle with Msoft support since thursday last week. For me, Its KB920958 and it can be forced to happen if the data folder is compressed (with explorer) and trigger it to happen immediately with a chkdsk c: /f and a reboot otherwise you have to wait for a while for it to go bad. We assume the compressoin is done by a LAZY process and sin't immediate. I don't think its the length as once the file is corrupt I have edited the contents to other values and replaced it without a problem, there must be a trigger byte sequence or something. If you look at the file with the 2000 resource kit dskprobe sector editor it complains about the length. Are you using compressed folders for yourr data cos I can't get it to fail otherwise and that seems to be my key trigger. We have disabled antivirus and gone through hoops to prove the point. Keith > Multi-posted - see your other posts. Please use cross-posting > if you wish to address several newsgroups. Multiposting wastes [quoted text clipped - 23 lines] > > > > Is there anyone else out there with the same problem? Hi Keith, Let's please try to compare commonalities: - Yes, on both servers those folders and parent folders have the compression flag on. - On both effected servers, the disks are defined as "dynamic" - They are software mirrored (using Windows 2000' Disk Management) - On both servers the files show all 0xDF characters (which shows up as the German special 's', looks a little like a B or a beta character.) - We run NetShield 4.5 with the 5.1.00 scan engine How does that compare with your environment? Andy at Argos.net > YES I have been in a losing battle with Msoft support since thursday last > week. [quoted text clipped - 45 lines] > > > > > > Is there anyone else out there with the same problem? The problem has now been seen on over 7 systems here in uk and usa, both win2000server and Pro across differing machine manufacturers so I have pretty well eliminated hard disk controllers, device drivers etc. Simply put if a folder is compressed and one of these (<4096 bytes) files is dropped into the compressed folder, then either after a period of time (hence my question of a "lazy" compression process) or by forcing with a chkdsk it will be corrupt immediately you reboot - this saves time waiting! I do have mirrored disks on the 2000 server but just standard basic disks on my 2000pros If the Kb920958 is removed, then no amount of file placement and chkdsk will cause the problem, all files already corrupt stay corrupt but no new damage occurs. If I confirm my compressed folder is good and then re add the kb920958 , followed by chkdsk /f etc the file corruption process occurs again. I have checked the files with 3 hex editors (Cygnus, win2kresource kit dskprobe and our own company proprietary viewer) The problem as far as I can see is KB920958 With Compressed folders and only CERTAIN files smaller than 4096 bytes (interesting match to cluster size perhaps?) I have dropped in text files of a k or so and they seem to be impervious. I have disabled all AV (was sophos), and other adaware checkers etc (that's the beauty of being in a test lab I can pull systems apart as I see fit) The most difficult question that our Msoft security case engineer wants answering is what other file types are affected. Until you popped up I was dealing with one of our own companies proprietary data storage types which should, as far as the O/S is concerned, be just a binary file. Once a file has gone bad then it never seems to "comes back", if I hexedit the 0xDF file content to another hex value the bad file doesn't seem to get made bad again (go back to 0xdf). This is why I was looking for a string of bytes as a trigger. Still , I don't have access to Msoft O/s Code and that's why I hoped raising a "case" with them would get it fixed. In answer to your other post (17:16) then At the moment, the solution for me is to Take off the kb920958 patch Go through storage and restore all bad files Wait for a fix. If the KB is necessary for you then I have not found corruption on a non compressed folder so maybe the option would be to remove compression, then restore the bad files In essence if you need the 920958 then don't enable compression. Keith > Hi Keith, > [quoted text clipped - 63 lines] > > > > > > > > Is there anyone else out there with the same problem? Hi Keith, okay, I will attempt to uninstall Kb920958 tonight! That's the best lead yet. I will also test the "compression" theory as soon as one of my customers reports another failure with one of their files. I definitely have seen at LEAST two files that are larger than 4096. However, both are very close to a MULTIPLE of 4096 (close to 8K in one case, in the other case 167,564 bytes = 163K). I inspected the files. In each case the beginning of the files are good, and ONLY the last 4000 or so bytes have the 0xDF byte pattern. I too created a small TXT file (just one line) and it never corrupted, while other files around it repeatedly did. I can say, that I have seen corruptions with the following file types: .JPG .GIF .SWF where the vast majority are small .JPG and .GIF thumbnails and so far I've only found few files that are larger. But that may be deceiving, because you simply may not notice the end of a larger file being corrupted, because the it "starts" out correctly! Best Regards, Andy > The problem has now been seen on over 7 systems here in uk and usa, both > win2000server and Pro across differing machine manufacturers so I have [quoted text clipped - 131 lines] > > > > > > > > > > Is there anyone else out there with the same problem? Hi Keith, you were "right on". Disabling compression on certain folders seem to fix it temporarily - and after uninstalling the hotfix last night on all servers with compressed disks I have had no more complaints thus far. Thank you for sharing your insight and for researching this so thoroughly. Let's see how long it takes Microsoft to take your bug report and proactively warn its customers that their hotfix can cause loss of data! Best Regards, Andy > The problem has now been seen on over 7 systems here in uk and usa, both > win2000server and Pro across differing machine manufacturers so I have [quoted text clipped - 131 lines] > > > > > > > > > > Is there anyone else out there with the same problem? PS: >> For me, Its KB920958 << Does that mean you have removed this Hotfix and your problems disappeared? > YES I have been in a losing battle with Msoft support since thursday last > week. [quoted text clipped - 45 lines] > > > > > > Is there anyone else out there with the same problem? I just removed KB920958. Its has not been more then an hour so its too early to tell. We are having problems with jpg files. Small thumbnails that we create is being corrupted. Heimir > PS: > [quoted text clipped - 51 lines] > > > > > > > > Is there anyone else out there with the same problem? 27 hours since I removed the update and we have not have any problems since. Heimir > I just removed KB920958. > Its has not been more then an hour so its too early to tell. [quoted text clipped - 3 lines] > > Heimir Out of curiosity, does Automatic Updates try to re-apply the patch? Did you have to turn that off? -jl > 27 hours since I removed the update and we have not have any problems since. > [quoted text clipped - 7 lines] > > > > Heimir I have auto update disabled as I need control of my environment but after I removed the patch it re downloaded it and notified me that it was there and waiting to be applied so I assume the answer is yes to your question. It is always a difficult compromise in an IT environment, do you want platform stability or up to date security, its a pity this has shown you can't always have both. As of this date I have still NO confirmation from the patch authors to acknowledge this as happening. So, either we are imagining this problem or this has bigger repercussions than we have discovered ourselves? > Out of curiosity, does Automatic Updates try to re-apply the patch? Did you > have to turn that off? [quoted text clipped - 12 lines] > > > > > > Heimir Hey Keith, thanks for the reply, I actually discovered this answer myself. I have 2 production environment that house most of my clients' files and to test I uncompressed one, and I uninstalled the patch from the other (which subsequently auto applied the patch again, hehe). It is a shame that I must choose between manual patching my application, and therefore violating my own security policy, or to uncompress many gigs worth of data, costing me 20-30% additional space now. I have to believe the lack of acknowledgement is out of embarassment or fear of the bug's effects. I have over 300 clients who upload images all the time and I've personally counted around 800 images (especially thumbnails) that kept getting corrupt during that week from Hell. Luckily, my business isn't file storage or graphic files. But if it was, this bug could of been catastrophic. And that's just a with a small business... If this isn't a Sev 1 bug within MS's system, I would be completedly shocked. My guess is the next round of patches will fix it and it will be done very quietly. -jl > I have auto update disabled as I need control of my environment but after I > removed the patch it re downloaded it and notified me that it was there and [quoted text clipped - 24 lines] > > > > > > > > Heimir Hi Keith. I'm posting over here as you requested. Right now I'm generating a list of all the files which get altered through the copy operation (I understand it can happen whenever *any* small file gets written or altered?). I'll be able to give you a list of filetypes and filesize limits as well. After that, I'll be trying to roll back the KB 920958 and see if that fixes my problem. Thanks! Frank > I have auto update disabled as I need control of my environment but after I > removed the patch it re downloaded it and notified me that it was there and [quoted text clipped - 24 lines] > > > > > > > > Heimir Ok, so this is what I've got. After checking some one million files of about 2500 different file types, I found evidence of corruption in some five thousand files, of about 150 different filetypes. That's about seven percent of all the filetypes listed. Contrary to what was seen earlier, I saw *no* 4k limit on file size. The largest file corrupted was an mpg file, some 41 mb in size. I also saw corruption in mp3's and other multimedia files (m4a, wav, etc.), and even gzipped and zipped files. conversely, even though a lot of jpegs were hit, out of all available jpegs, only about 3% were corrupted. Now I'll be rolling back that patch, and confirm if the problem goes away for all cases. > I have auto update disabled as I need control of my environment but after I > removed the patch it re downloaded it and notified me that it was there and [quoted text clipped - 24 lines] > > > > > > > > Heimir im sending zip files to a customer's windows ftp server. they've been complaining intermittenly since 2006.08.09, that the zip files we're sending are occasionally corrupted. upon inspection, the bad versions are ending on 0xdfs. when i create a hexdump of both the bad and good copy and then diff them: --- > 0036000 dfdf dfdf dfdf dfdf dfdf dfdf dfdf dfdf > * > 0036ec0 dfdf dfdf dfdf > 27 hours since I removed the update and we have not have any problems since. > [quoted text clipped - 7 lines] > > > > Heimir I saw the same thing with my files. removing the kb patch removed the problem. > im sending zip files to a customer's windows ftp server. they've been > complaining intermittenly since 2006.08.09, that the zip files we're [quoted text clipped - 19 lines] > > > > > > Heimir We are experiencing something very similar at my company, but it's our customers that are having the problem, not us. Our product is a Windows-based product that is currently being supported back to Windows 98. One of the things we do is save information (data, graphs, etc.) using OLE structured storage. We compress some of the larger data streams using a third party package. Since Sept 5, 2006 our Tech Support staff has received three corrupted files which, upon further review by the Software Development team, show the presence of 0xdf at the end of the compressed stream. Some information about the occurrence and amount of 0xdf: Corruption 1: Compressed data stream size - 214749 bytes 0xdf starts at byte 210944 Total 0xdf bytes - 3805 Corruption 2: Compressed data stream size - 901867 bytes 0xdf starts at byte 898048 Total 0xdf bytes - 3819 Corruption 3: Compressed data stream size - 98889 bytes 0xdf starts at byte 95232 Total 0xdf bytes - 3657 This is devastating to our customers because subsequent attempts by our software to open the saved file fail because the "decompression" algorithm fails. We are currently gathering information from the affected customers to see if hot fix KB920958 is a possibility. > YES I have been in a losing battle with Msoft support since thursday last > week. [quoted text clipped - 45 lines] > > > > > > Is there anyone else out there with the same problem? Microsoft Support confirmed Hotfix 920958 is bad: "We are aware the issue you are experiencing. A corresponding bugcheck request is currently open, and the develop team is working on this issue. However, the hotfix for this issue is not ready. 0xDF is the data pattern that NTFS returns when it has problem to decompress the file (e.g.. the compression fragments are corrupted and can't be decompressed). Based on my research, the actual raw data on the disk is not changed, it shows as 0xDF because the system cannot decompress the file and display the data correctly. So the corrupt is not permanent. Further more, the issue only occurs on files which containing Hexadecimal codes." " Further more, the issue only occurs on files which containing Hexadecimal codes." This must be why you (or others?) couldnt get a text file to be corrupted. Yes. Although we only tried very small text files - so another theory was that it was just too small since it seemed to always effect files close to 4096 bytes or a multiple thereof. (Of course, that was "anecdotal evidence" and may or may not have been a conincident.) > " Further more, the issue only occurs on files which containing > Hexadecimal codes." > > This must be why you (or others?) couldnt get a text file to be > corrupted. > " Further more, the issue only occurs on files which containing > Hexadecimal codes." If I have a look inside my files, all of them contain hexadecimal codes :-) It seems as if only non-compressible files with a (filesize modulo 4096) of about 3600 to 4095 became victims of the fatal bug. The list of my damaged files contains mostly JPG-Files and archives. In detail one 7z-files, eight exe-files (packed?), one bz2-file, five gz-files, one tif-file, 116 .jpg-files and nine zip-files were partially overwritten with the ßßßßßßß-sequence. I cannot believe that the "raw" file on the disk is unharmed, as stated by the microsoft employee above, for even Ubuntu Linux, when bootet from CD at the same machine, read the same garbage out of the files as Windows 2000 either with the patch deinstalled or installed. In microsoft.public.win2000.file_system wrote: >> " Further more, the issue only occurs on files which containing >> Hexadecimal codes." > > If I have a look inside my files, all of them contain > hexadecimal codes >:-) Yep, a "good one". > It seems as if only non-compressible files with a (filesize > modulo 4096) of about 3600 to 4095 became victims of the fatal > bug. The list of my damaged files contains mostly JPG-Files and > archives. In detail one 7z-files, eight exe-files (packed?), one > bz2-file, five gz-files, one tif-file, 116 .jpg-files and nine > zip-files were partially overwritten with the ???????-sequence. In my testing here I found no existing (pre-KB920958) files suffered corruption within +C directories. Setting -C succeeded in decompressing all those files without damaging any. I tested with a thousand random GIF and JPG files copying them into a test directory set +C. Approximately 12% became corrupted. ALL of the files damaged were in a range of <even cluster value here>.89 to <even cluster value here>.99 (W2K NTFS default - 4096 bytes) > I cannot believe that the "raw" file on the disk is unharmed, as Neither can I (for new files). > stated by the microsoft employee above, for even Ubuntu Linux, > when bootet from CD at the same machine, read the same garbage > out of the files as Windows 2000 either with the patch > deinstalled or installed. I suspect the files are corrupted (post KB920958) when copied into a +C location. End of story after that. > In microsoft.public.win2000.file_system wrote: >> It seems as if only non-compressible files with a (filesize >> modulo 4096) of about 3600 to 4095 became victims of the fatal [quoted text clipped - 26 lines] > I suspect the files are corrupted (post KB920958) when copied into > a +C location. End of story after that. I have a compressed, corrupted binary file that is damaged on disk but is OK in a backup (which is also compressed via NTFS). After removing KB920958 the original is still damaged if read as compressed or decompressed; confirming that the raw data is indeed damaged. Cheers, Adam Piggott, Proprietor, Proactive Services (Computing). http://www.proactiveservices.co.uk/ Please replace dot invalid with dot uk to email me. Apply personally for PGP public key. In microsoft.public.win2000.file_system =?Utf-8?B?QW5keSBTY2htaWR0?= wrote: > Hi, > > I have two different servers (different models and manufacturing > years), but both of them running Windows 2000 server with > software mirrored NTFS drives, which suddenly corrupt small > image thumbnail files after an hour or two. [ ] Sep 15, 2006 http://support.microsoft.com/kb/920958 (MS06-049) Article ID : 920958 Last Review : September 15, 2006 Revision : 2.0 MS has updated the article, to acknowledge the problem with potential file corruption. While the KB920958 file has not yet been re-released, MS is offering a Hot Fix as detailed in http://support.microsoft.com/kb/925308 ========================================== To resolve this problem immediately, contact Microsoft Product Support Services to obtain the hotfix. For a complete list of Microsoft Product Support Services phone numbers and information about support costs, visit the following Microsoft Web site: ========================================== US anyway: 1-800-936-4900, opt. 1 and "Article ID : 925308" "Compressed files that are larger than 4 kilobytes may be corrupted when you create or update the files" "Windows 2000" Etc. No charge. Note that the original fix, Windows2000-KB920958-x86-ENU.EXE will be updated eventually. This Hot Fix is subject to further testing and so on... In microsoft.public.win2000.file_system Mark V wrote: > In microsoft.public.win2000.file_system > =?Utf-8?B?QW5keSBTY2htaWR0?= wrote: [quoted text clipped - 6 lines] >> image thumbnail files after an hour or two. > [ ] > Sep 15, 2006 > [quoted text clipped - 16 lines] > about support costs, visit the following Microsoft Web site: > ========================================== [ ] Following up: The HotFix (HOT-FIX_287398_ENU_i386_zip.exe for Windows2000-KB925308-x86-ENU.EXE) appears here to have entirely resolved the file corruption issue. No test files copied into, out of, or edited in-place (in +C location) suffered any damage whatsoever. "Test files" includes files known to corrupt pre- HotFix. Recall that in all probability any files damaged pre- HotFix will remain damaged (not specifically tested here). Installed HotFix on W2K Pro, NTFS, SP4+ and without uninstalling Windows2000-KB920958-x86-ENU.EXE (MS06-049) FWIW and YMMV > While the KB920958 file has not yet been re-released, MS is > offering a Hot Fix as detailed in > http://support.microsoft.com/kb/925308 As a FYI - I have written a scanner that will list files that seem to be corrupted by KB920958. It's available from http://marc.kupper.googlepages.com/scandf This utility can't repair your files though it's possible enough information was written to disk that Microsoft will be able to issue an update that allows for reading the original file data. Marc In microsoft.public.win2000.file_system Marc Kupper wrote: >> While the KB920958 file has not yet been re-released, MS is >> offering a Hot Fix as detailed in [quoted text clipped - 3 lines] > seem to be corrupted by KB920958. It's available from > http://marc.kupper.googlepages.com/scandf In case it is not otherwise known, that mentioned HotFix was publicly released (same files) on the 20th as KB-925308 http://support.microsoft.com/kb/925308/ > This utility can't repair your files though it's possible enough > information was written to disk that Microsoft will be able to > issue an update that allows for reading the original file data. I doubt that is possible. Damaged files were literally stored in that condition in the file system...data irreparably lost, restore from clean backup. :( Earlier I had written: > > ... it's possible enough > > information was written to disk that Microsoft will be able to > > issue an update that allows for reading the original file data. Mark V replied: > I doubt that is possible. Damaged files were literally stored in > that condition in the file system...data irreparably lost, restore > from clean backup. :( I'd tend to agree and late last night had another thought in that it's possible a data block for the last block was not allocated/written to at all. I'm basing this thought on that it's always the last block that's lost plus a comment on slashdot iirc where someone got a file size mismatch error when they tried to load up a corrupted file using some sort of raw disk editor (where they'd see whatever is actually written to disk). One of the first things I did though when I realized files were getting corrupted was to run chkdsk which did not find any problems. Initially I had been thinking that *something* was written to disk but that its structure was not recognized by the decompression code which then spun out the DF bytes in it's place. Marc In microsoft.public.win2000.file_system =?Utf-8?B?QW5keSBTY2htaWR0?= wrote: > Hi, > > I have two different servers (different models and manufacturing > years), but both of them running Windows 2000 server with > software mirrored NTFS drives, which suddenly corrupt small > image thumbnail files after an hour or two. [ ] Microsoft Security Bulletin MS06-049 Vulnerability in Windows Kernel Could Result in Elevation of Privilege (920958) Published: August 8, 2006 | Updated: September 26, 2006 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ================== excerpt ====================== V2.0 (September 26, 2006): The update has been revised and re- released for Microsoft Windows 2000 Service Pack 4 to address issues identified in Microsoft Knowledge Base Article 920958. ===================================================== http://www.microsoft.com/technet/security/Bulletin/MS06-049.mspx This is the patch (v1) for W2K that caused corrupt files in on-disk Compressed files. First MS issued a HotFix for W2K 2006-09-15 Then issued those same files via KB-925308 2006-06-20 And finally they have incorporated the files for W2K back into the "049" package for W2K. W2K users should apply one of the "corrected files" packages if they ever installed MS06-049, (920958) version 1. W2K users that store Security pkg. files locally should update their stored copy to "version 2". Windows2000-KB920958-v2-x86-ENU.EXE 1,603,560 bytes MD5: 09f6b7733eca534b2620549e1e184f7d All three sources of "corrected files" contain the same 5 updated files. > Hi, > [quoted text clipped - 15 lines] > > Is there anyone else out there with the same problem? |
Reply to this Message |
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.