WIMBoot: A Still-Born Compression Technology, Easily Defeated by Decades old NTFS Compression?

Discussion in 'Windows 8' started by zipmagic, Apr 2, 2015.

?

Would you use compression to increase the available storage on your tablet PC or expensive SSD?

  1. I'd use 20-decades old NTFS compression, known to be reliable.

    3 vote(s)
    60.0%
  2. I'd use brand-new WIMBoot compression, to compress more.

    0 vote(s)
    0.0%
  3. I'd use third party tooling, known to be friendlier.

    2 vote(s)
    40.0%
  1. zipmagic

    zipmagic Pen Pal - Newbie

    Messages:
    98
    Likes Received:
    12
    Trophy Points:
    16
    I have made some very interesting discoveries regarding WIMBoot technology. Please take a look at the attached Excel and the relevant screenshots. To emphasize the effect of compression on the amount of free space available, all tests were conducted on a physical, 16 GB eMMC tablet. This is also a primary showcase for WIMBoot technology, as Microsoft introduced WIMBoot specifically to solve space shortage issues on small 16 GB tablets.

    The results are absolutely shocking and very news-worthy: WIMBoot does NOT manage to make a substantial difference above two decades old NTFS compression. In fact, NTFS compression defeats WIMBoot by a large margin on a fully loaded tablet, and is only narrowly defeated by WIMBoot on a fully empty, brand-new tablet. The numbers in the Excel sheet speak for themselves, load it up.

    Do you understand what this means? Microsoft engineers built WIMBoot in vain. While it is solid compression technology, and it compresses a lot better than two-decades old NTFS compression; it is rendered practically useless because of what I have termed the “space bleed” problem, and the utter lack of tooling around WIMBoot that would be required to address the “space bleed” problem.

    What is this “space bleed” issue? When you compress a disk with WIMBoot, any existing data on disk cannot be effectively deleted or updated. Your entire data is now inside a single ‘backing’ WIM file, where deleting a file simply marks it ‘deleted’ and does not recover space from the WIM. It’s even worse when you modify a file inside a WIM: A duplicate, uncompressed copy is extracted to disk, so your modifications can be saved.

    This is why, on a tablet loaded with a modest number of Windows Updates, you rapidly run out of space. Updated files consume almost twice the storage on disk – once compressed, inside the ‘backing’ WIM file, and twice on disk, fully uncompressed. Any compression savings to be had by WIMBoot are lost rapidly, allowing 20-years old NTFS compression to eventually exceed WIMBoot in space savings, despite being 20 years old!

    To reproduce these results, you will need to use an NTFS compressor like ZIPmagic DriveSpace 5.1, which compresses your entire disk safely and securely – if you try to use Windows File Explorer to apply NTFS compression, it will fail to compress a majority of files on disk, simply because it lacks the necessary access rights to protected program files and system folders.

    WIMBoot is very sound technology at its core, but lacks the proper GUI tooling to make it usable. Using a WIMBoot compressor like ZIPmagic DoubleSpace 2, you can recompress a disk to recover space from deleted/updated files, and exclude folders which you know are frequently updated from compression altogether. Please check the screenshots with DoubleSpace 2 recompression applied to the tablet, they show the most free space on disk.

    In conclusion, Microsoft may have as well fixed Windows File Explorer’s NTFS compression bug in Windows 8.1, instead of investing in WIMBoot, which is utterly unusable without third party tools to back it. Please take a look at the screenshots below.

    Generic 16 GB eMMC tablet with default OEM-supplied WIMBoot:
    baseline.PNG

    Generic 16 GB eMMC tablet recompressed with DoubleSpace 2 WIMBoot/MaxSpace setting:
    baseline-maxspace.PNG

    Generic 16 GB eMMC tablet recompressed with DoubleSpace 2 WIMBoot/MaxSpeed setting:
    baseline-maxspeed.PNG

    Generic 16 GB eMMC tablet decompressed and WIMBoot removed with DoubleSpace 2, then recompressed with DriveSpace 5.1 using NTFS compression:
    baseline-ntfs.PNG

    Generic 16 GB eMMC tablet, first loaded up with the free Office 365 offer found in the tablet, and then updated with all Windows Updates; with default OEM-supplied WIMBoot:
    baseline.PNG

    Generic 16 GB eMMC tablet, first loaded up with the free Office 365 offer found in the tablet, and then updated with all Windows Updates; then recompressed with DoubleSpace 2 WIMBoot/MaxSpace setting:
    baseline-maxspace.PNG

    Generic 16 GB eMMC tablet, first loaded up with the free Office 365 offer found in the tablet, and then updated with all Windows Updates; then recompressed with DoubleSpace 2 WIMBoot/MaxSpeed setting:
    baseline-maxspeed.PNG

    Generic 16 GB eMMC tablet, first loaded up with the free Office 365 offer found in the tablet, and then updated with all Windows Updates; then decompressed and WIMBoot removed with DoubleSpace 2, then recompressed with DriveSpace 5.1 using NTFS compression:
    baseline-ntfs.PNG
     

    Attached Files:

  2. jnjroach

    jnjroach Technology Strategist Super Moderator

    Messages:
    3,858
    Likes Received:
    1,161
    Trophy Points:
    181
    WIMBoot was design specifically for one use case and is being completely deprecated in Windows 10 which is using a different mode and approach.
     
  3. zipmagic

    zipmagic Pen Pal - Newbie

    Messages:
    98
    Likes Received:
    12
    Trophy Points:
    16
    I felt my study was interesting especially because this appears to be the exact use case Microsoft optimized for: Small 16 GB toy tablet. Yet even with this designed-for use case, the space bleed problems rendered the scenario virtually unworkable - with only 1.6 GB free space after installing just Office 365 from the offer that came with the tablet, and Windows Updates.

    I have been trying to find information on the new compression in Windows 10, it sounds interesting. I wonder if it is based on NTFS, WIMBoot, or a completely new kernel-side technology. It is also a big question whether the usability will be plagued by the same type of problems that currently plague NTFS and WIMBoot today.

    Make no mistake - I think Microsoft has done a terrific job with both NTFS compression, and WIMBoot compression, as it is. Its just the usability issues that seem to cause problems with both technologies.

    If it were up to me, I would vote for Microsoft implementing various stronger compression grades with NTFS compression. NTFS's ability to recompress data while saving files is exceptionally user friendly, and I'm sure today's processors, which are probably about a 1,000 times faster than those available when NTFS was first designed, could cope with the increased load.
     
  4. Mikerman

    Mikerman Scribbler - Standard Member Senior Member

    Messages:
    1,293
    Likes Received:
    126
    Trophy Points:
    81
    zipmagic, you do some of the most interesting, informative, educational advertising. ;)
     
    jhoff80 likes this.
  5. andy_sym

    andy_sym Pen Pal - Newbie

    Messages:
    27
    Likes Received:
    2
    Trophy Points:
    6
    As an average geek Windows enthusiast I'm still unable to quite get my head around the steps required to create my own WIMBoot successfully so I used DoubleSpace on my tablet during the beta stages and it worked great and freed up GB's of space in about 30 mins from start to finish.

    I also find the background to these various space saving / compression methods interesting and forgot all about Windows NT compression but was unaware Windows 10 will use something different again although from my own virtual instance of Windows 10 unless something drastic changes on that I won't be moving from Windows 8 even if it only uses 1GB space!
     
  6. jhoff80

    jhoff80 Scribbler - Standard Member Senior Member

    Messages:
    3,178
    Likes Received:
    872
    Trophy Points:
    181
  7. zipmagic

    zipmagic Pen Pal - Newbie

    Messages:
    98
    Likes Received:
    12
    Trophy Points:
    16
    Sounds dangerous from a system corruption/anti-virus perspective. If the original files are corrupted, there's not going to be many options moving forward - short of a fresh re-download of affected components to replace them, or a full backup image like WIMBoot (which they've already said is off the table).

    I'm also curious about the minimum viable hardware requirement. The 16 GB eMMC tablet I was testing on certainly is a toy-class tablet, very limited capabilities. However it was able to handle all the compression tasks thrown at it just fine, even the strongest grade of WIMBoot/MaxSpace compression caused no issues or hiccups.

    It kind of makes sense - NTFS compression was invented two decades ago, when the best CPU was a 33 MHz 486 DX. (Later on when they did clock doubling on these devices they released 50 MHz DX2, and even a 100 MHz DX4 [I believe those were based on the 25 MHz variant of the 486 CPU].) But anyways, CPU's are now about a 1,000 times faster - so yeah, there could be no performance concerns with any kind of NTFS compression.

    The fact that the toy tablet performed admirably even with WIMBoot/MaxSpace compression attests to the engineering prowess of the Microsoft team. WIMBoot is very recent, so it could have proven very taxing for this hardware. I was expecting it to prove so, I was pleasantly surprised when it did not.
     
  8. zipmagic

    zipmagic Pen Pal - Newbie

    Messages:
    98
    Likes Received:
    12
    Trophy Points:
    16
    Several hours ago, I launched ZIPmagic 12.8, with the brand-new DriveSpace 6. DriveSpace 6 adds the following improvements on top of the previous version DriveSpace 5.1:

    1) DriveSpace no longer compresses files using NTFS/LZNT1 when those files are larger than 32 GB (this could cause issues writing to those files, because NTFS does not support such large files for dynamic writing – although straight reads and decompression works).

    2) DriveSpace supports all new compression formats introduced with Windows 10 natively. These new compression formats bring the compressing strength of WIMBoot, with virtually none of the hassles of the original WIMboot release. To solve the space bleed problem discussed earlier, Microsoft has converted compression in Windows 10 to being on a per-file basis, instead of a per-image basis. This cures the space bleed problem entirely on deleted files, and the only thing needed to address space bleeding on updated files is to periodically recompress your disk. Compressing or recompressing a disk no longer requires Windows to be taken offline, just like NTFS compression. Everything can happen on-line, which is especially a great boon for server operating systems being connected to via Remote Desktop – they too can now benefit from the terrific space savings introduced with WIMBoot.

    3) DriveSpace can go from LZNT (old NTFS) compression to any new Windows 10 (WIM2) compression. It can also go in the reverse direction (not supported by compact.exe shipping with Windows). All files will be dynamically decompressed and recompressed as necessary. There are four new grades of Windows 10 (WIM2) compression available: XPRESS 4KB, XPRESS 8KB, XPRESS 16KB, and LZX. These are identical to the compression settings currently on offer with DoubleSpace. Even though DoubleSpace compresses a bit better than DriveSpace when you use the open source WIMLIB library instead of the official Microsoft library, the same basic ballpark compression grades are now unlocked for DriveSpace as well.

    4) DriveSpace recommends the utilization of the exact number of CPU cores for XPRESS based WIM2 compression, and two thirds of the exact number of available CPU cores for LZX based WIM2 compression. These are both unlike NTFS, which can handle up to twice the number of available CPU cores during compression. Using such a high core count with WIM2 is contra-indicated, because of the modern and taxing nature of new WIM2 compression, unlike NTFS compression which is two decades old and was therefore exceedingly efficient on modern CPUs, being limited only by the underlying storage hardware. You should not use more than the recommended number of CPU cores by DriveSpace on any system, including an SSD system; because the CPU will become the bottleneck, no matter how fast your underlying SSD is. Open the Task Manager and keep an eye on your CPU and HDD utilization if you are interested, and want to experiment with different core utilization settings. Both should be as close to 100% as possible, or you are wasting computing resources at either the CPU end or the HDD end.

    5) DriveSpace does not compress files larger than 4 GB using the new WIM2 compression, which still supports only a maximum of 4 GB file size for compression. It will automatically compress such files using LZNT1/NTFS compression instead.

    6) An obscure bug with the shell namespace extension (file archiving component) has been fixed (this is not related to DriveSpace but the archiving components of the ZIPmagic suite at large).

    7) DoubleSpace now recommends you run DriveSpace on Windows 10, unless your Windows 10 boot disk was already WIMBoot’ed (using WIMBoot version 1). In this scenario, you could use DoubleSpace to uncompress your disk from the legacy WIMBoot format, and recompress it using the new WIM2 format. This will get you almost all of the original space savings of the original WIMBoot, with virtually none of the space bleed hassles. I am still researching how much exactly the space differential is, and whether it may still be worth WIMBoot’ing your disk. Early results indicate about a 15%-20% improvement with WIMBoot versus the new WIM2, but these results have not been yet validated at a large scale (testing is ongoing). You will probably still want and need to use DoubleSpace if you absolutely want to maximize disk storage, and aren't concerned about the need for periodic offline disk recompression, and you have an Undo Disk available. Otherwise, especially for remote servers, or for users without an Undo Disk, and the capability to take Windows offline for recompression operations; DriveSpace just makes more sense.

    8) Since DriveSpace is a major version upgrade, the serial algorithm has also been updated. ZIPmagic users who purchased in the last 30 days are entitled to a free upgrade. ZIPmagic users who are current on their maintenance plans are also entitled to a free upgrade. Any other ZIPmagic user may contact ZIPmagic to receive a 70% discount coupon to upgrade to this version.

    Please note that it is also possible to get a glimpse of some of the compression savings of DriveSpace for absolutely free from the Windows 10 command line, running the following from an elevated command line window:

    compact /compactos:always

    This approach of course lacks several DriveSpace benefits:

    a) Multi-threaded processing for the quickest disk compression/re-compression/uncompression times.
    b) Automatic processing of files larger than 4 GB with LZNT1 compression (compact fails on them with WIM2 compression).
    c) Automatic exclusion of files larger than 32 GB from all compression (compact would succeed with LZNT1, but apps would fail upon trying to write to the file).
    d) Ability to downgrade the compression setting without having to decompress the entire disk first, either to a lower WIM2 setting, or to the lowest LZNT1 (NTFS) setting.
    e) Ability to process the entire disk with LZNT1 (NTFS) compression - compact still fails to do so with the command line above.
    f) Specifying file-exclusions on a file-extension basis (ideal for servers hosting Microsoft SQL Server files - please note that this behavior is also unavailable with ZIPmagic Consumer Edition, and is only available with ZIPmagic Server Edition).

    However, you would still gain some space with the above and not "bleed" extra free space when data is extracted to disk for any reason (such as patches); whereas in the original WIMBoot iteration, you absolutely would be unable to live without a tool like DoubleSpace in the long term.
     
    Last edited: Aug 8, 2015
  9. zipmagic

    zipmagic Pen Pal - Newbie

    Messages:
    98
    Likes Received:
    12
    Trophy Points:
    16
    Well, the plot thickens somewhat. After testing both compact.exe, DriveSpace 6, and DoubleSpace 2 on clean machine installs of Windows 10 32-bit and 64-bit, build 10,240, there's some unexpected but consistent and statistically significant variance in results.

    Here's 32 bit Windows 10 compressed using DriveSpace 6, with LZX compression:
    Windows 10 x86 Build 10041-DriveSpace 6 LZX compressed.png
    Here's the same operating system compressed using DoubleSpace 2 with LZX (MaxSpace) compression:
    Windows 10 x86 Build 10041-DoubleSpace 2 at MaxSpace.png
    Summary for 32-bit Windows 10:
    DoubleSpace 2 compresses ~630 MB better than DriveSpace 6.

    Here's 32 bit Windows 10 compressed using DriveSpace 6, with LZX compression:
    Windows 10 x64 Build 10041-DriveSpace 6 LZX compressed.png
    Here's the same operating system compressed using DoubleSpace 2 with LZX (MaxSpace) compression:
    Windows 10 x64 Build 10041-DoubleSpace 2 at MaxSpace.png
    Summary for 64-bit Windows 10:
    DoubleSpace 2 compresses ~980 MB better than DriveSpace 6.

    It seems on average, the old, space-bleeding version of WIMBoot, compresses about 0.75 GB better than the new non-space-bleeding version. This is interesting, and if these results scale to operating systems populated with many apps and data, could be a reason to keep using space-bleeding WIMBoot, if your space requirements dictate so. Possible reasons for this include:

    - Better compression in DoubleSpace 2 with the open-source WIMLIB library compared to WIMGAPI (Microsoft library).
    - Ability to compress all Windows files in exclusive mode while the system is not running.
     

    Attached Files:

    Last edited: Aug 8, 2015
  10. zipmagic

    zipmagic Pen Pal - Newbie

    Messages:
    98
    Likes Received:
    12
    Trophy Points:
    16
    Please see the new System Refresh option in DoubleSpace 2:

    systemrefresh.png

    This option restores your system to the same state it was in the last time you compressed it.

    It does not require an Undo Disk, and it runs very fast. After rebooting, it just deletes all files on your disk (except those in excluded folders, and those that must remain like the Windows Recovery Environment), and then re-applies your WIM (which it does not delete either, of course) in compressed mode.

    To be frank, it was simpler to implement this behavior in this fashion; rather than having to interface with Microsoft’s one-click reset tool layering. However the net result is the same: a one-click reset experience.

    Hoping this helps DoubleSpace aficionados everywhere :)
     
Loading...

Share This Page