Update command in wimlib-imagex (Windows)

maxpat78
2013-04-13
2013-05-22
1 2 > >> (Page 1 of 2)
  • maxpat78

    maxpat78 - 2013-04-13

    Actually, wimlib-imagex on Windows can't mount an image in rw mode, so the image can't get updated, with addition/remotion of files and dirs.

    But would it be easy to provide an "update" command, like in my ImagePyX?

     
  • synchronicity

    synchronicity - 2013-04-13

    I'm planning to do that, but it's low on my priority list. The problem with this sort of thing is that basically you end up having to re-implement well-known operations (add, recursive add, delete, recursive delete, extract, recursive extract...) that would otherwise "just work" if the filesystem was mounted read-write. It would be much more useful to implement libfuse for Windows and use that instead; unfortunately this may be very difficult or even impossible due to limitations and eccentricities in the Windows driver and filesystem models.

     
  • maxpat78

    maxpat78 - 2013-04-13

    ...and for 64-bit envs you'd have to sign your driver... :(

    Hint: now I'm exploring BackupRead/WIN32_STREAM_ID to capture ADS before Vista. Also, my new code captures and restores "as expected" (=honoring FLAG_FIX_RP) (sym)links both in Windows and Linux.

     
    Last edit: maxpat78 2013-04-13
  • synchronicity

    synchronicity - 2013-04-13

    I will consider using BackupRead(), which is documented as being for tape backups, but the insanity only goes so far--- when Windows doesn't even offer a properly documented API to read data from its own filesystem I'm not sure I should bother.

     
  • maxpat78

    maxpat78 - 2013-04-14

    Yes, despite of its name, it is a sort of "extended ReadFile", which can read SDs and ADSs in one pass (the main unnamed $DATA stream among them)... for backup, or other, purposes.

     
  • maxpat78

    maxpat78 - 2013-04-15

    In the case, my v0.29 released just now has the above XP support

     
  • synchronicity

    synchronicity - 2013-05-18

    Hi,

    Primarily to work around the lack of mounting support on Windows, wimlib-imagex version 1.4.0 does include "extract" and "update" commands. The update functionality should be more flexible than the support in ImagePyX, since it allows you to add, delete, and/or rename any number of files or directories, with various options, in a single wimlib-imagex command. Let me know if you have a chance to test it.

    I have not implemented the BackupRead trick you mentioned above.

     
  • maxpat78

    maxpat78 - 2013-05-20

    Hello to you, I've downloaded v.1.4 yesterday and was just about to do some tests (I was thinking myself about improving ImagePyX with delete/rename functions and/or making a simpler port in C++, but time is very short at the moment).

    Also, I've just noticed I can't update a backup of mine due to the very long file name I talked you about in a previous post.

    Since you actually don't trust the \\?\ trick, you could at least consider accessing the LFN object by ITS SHORT NAME instead...

     
  • maxpat78

    maxpat78 - 2013-05-20

    Ah, and, yes, of course I suggest you to avoid aborting the operation, by simply skipping the file with the longer file name!

     
  • synchronicity

    synchronicity - 2013-05-20

    Hi, thanks for bringing this up again--- I might be able to "fix" it in the next release by simply translating the add sources and extract targets into \\?\-prefixed full paths, then making sure any added path separators are backslashes rather than forward slashes. However, this is all very frustrating because it's very much Microsoft's fault that their OS is by default incompatible with its own default filesystem. I will also need to individually verify that each of the following functions work with the special path format (I checked the "documentation" for each of them and there is no mention of it):

    OpenEncryptedFileRawW()
    GetFileSecurityW()
    FindFirstStreamW()
    EncryptFileW()
    DecryptFileW()
    CreateHardLinkW()

    In the case of OpenEncryptedFileRawW() the documentation additionally states that it does not support characters outside the Windows character set (ANSI code page), although that's probably not actually true since it's path parameter must be given in UTF16-LE, so go figure.

    Thanks!

     
  • synchronicity

    synchronicity - 2013-05-20

    I think I have the extraction of long paths working now, although I was amused that neither cmd.exe nor Windows Explorer allowed me to access my extracted files. Perhaps that's a feature to look forwards to in Windows 2095.

     
  • maxpat78

    maxpat78 - 2013-05-21

    Thanks to you!

    Very strange what you said about CMD & Explorer: as far as I can remember, when I was experiencing problems with ffmpeg and "longer" pathnames, the access to them with CMD and 7-Zip went fine (Win7 x32, NTFS): what Win version are you using?

     
  • synchronicity

    synchronicity - 2013-05-21

    I was using Windows 7. Nothing happened when I clicked on the folder (whose full path exceeded MAX_PATH) in Explorer, and cmd.exe refused to enter a working directory with name longer than MAX_PATH. Nor would the "rd" command remove the directory tree. So obviously Windows does not have good support for its own filesystem. 7-Zip probably uses the \\?\ path hack which is why it seemed to work.

    If you want to test out long file support or anything else before I officially make a 1.4.1 release I posted the latest binaries on the files page (wimlib-1.4.1-windows-i686-BETA-bin.zip).

     
  • maxpat78

    maxpat78 - 2013-05-21

    Hi, I've tried your Win32 1.4.1-beta: LFN capturing works, but I've had an issue restoring the image to non-NTFS volumes, probably due to poor ADS handling (wimlib-imagex failed to create a "Thumbs.db:encryptable" file: it's the first time I see a stream with name "encryptable"; all I can say is that file can't be encrypted, since Win8 basic doesn't support that).

    Besides that, if you do:
    wimlib-imagex capture DIR x.wim
    wimlib-imagex append DIR x.wim

    it stops the appending op with the error "There is already an image name DIR in the WIM".
    Perhaps you should change the image name induction mechanism, or not to guess the image name at all, like my ImagePyX does.
    Also, I'm not completely sure a duplicate image name has to be prohibited (only the index seems the real unique key for images).

    Finally, I was pleased to test the "batch" delete/add/rename commands for the update op: they worked fine, but I noticed delete and rename print no message -- perhaps they should tell the user what is happening!

     
  • maxpat78

    maxpat78 - 2013-05-21

    About the LFN support in Windows 8-x64 NTFS, I played a bit with Python and experienced issues myself.
    In fact, you have to prepend the \\?\ when the dir name is >244 chars (not 260 as I expected).
    As expected, the single dir name can't exceed 255 chars. But, if you create a second nested subdir with a 255-chars name, CMD can't access nor DIR it; Explorer doesn't recognize it as a directory and can't open it; 7zFM can open it and seems able to copy a file inside it: but then it can't operate with that file... (Python, instead, runs fine if you prepend the magic string before opening the contained file).

     
  • synchronicity

    synchronicity - 2013-05-21

    Hi, I've tried your Win32 1.4.1-beta: LFN capturing works, but I've had an issue restoring the image to non-NTFS volumes, probably due to poor ADS handling (wimlib-imagex failed to create a "Thumbs.db:encryptable" file: it's the first time I see a stream with name "encryptable"; all I can say is that file can't be encrypted, since Win8 basic doesn't support that).

    Thanks for finding this. It was actually caused by a place in my code thinking that \\?\-prefixed paths were relative to the current drive (as they don't "begin" with a drive letter). So it was just a bug with using this new crazy path format. Otherwise, the code is supposed to use GetVolumeInformation() to get the features of the destination filesystem and only extract data that is supported (with warnings when appropriate). I'm just about to commit a fix for this.

    Besides that, if you do:
    wimlib-imagex capture DIR x.wim
    wimlib-imagex append DIR x.wim
    it stops the appending op with the error "There is already an image name DIR in the WIM".
    Perhaps you should change the image name induction mechanism, or not to guess the image name at all, like my ImagePyX does.
    Also, I'm not completely sure a duplicate image name has to be prohibited (only the index seems the real unique key for images).

    I'm not convinced this should be changed, for a couple reasons. Although you're correct that the image index is the primary way to identify images, they can also be identified by name, so allowing multiple identically named images would cause confusion. Furthermore, Microsoft's ImageX itself enforces this constraint (when appending at least). The only difference with wimlib-imagex is that wimlib-imagex doesn't require you to specify an explicit name, as it takes the name of the source directory as the default. One possible improvement, though, would be to have `wimlib-imagex append' suggest the syntax to provide a different image name in the event of an image name conflict. Otherwise I suppose the user might think that it's not possible to append the image at all.

    Finally, I was pleased to test the "batch" delete/add/rename commands for the update op: they worked fine, but I noticed delete and rename print no message -- perhaps they should tell the user what is happening!

    Yes, I can add progress callback messages for these operations in the library, so wimlib-imagex can print when they occur. Note, however, that delete and rename are purely in-memory operations (at least, before actually performing the overwrite, which happens after all the commands have been processed) and therefore will be completed very quickly.

    Anyway, thanks for testing --- it's very useful!

     
  • synchronicity

    synchronicity - 2013-05-21

    In fact, you have to prepend the \?\ when the dir name is >244 chars (not 260 as I expected).

    I remember reading somewhere that there is some quirk where the directory path always has to be short enough so that it can contain an 8+3 filename without exceeding MAX_PATH. The current code in wimlib always uses the \\?\ prefix though, so this should not be an issue.

     
  • maxpat78

    maxpat78 - 2013-05-21

    I've just fixed a bug in my ImagePyX, related to that :encryptable ADS.
    My code tried to make the File resource for the ADS, without detecting that its size is (incredibly) of zero bytes!

     
  • maxpat78

    maxpat78 - 2013-05-21

    P.S. I've been considering long ago if and how to resolve the image name conflicts in my ImagePyX, too, but I think that allowing unnamed images is a better solution (ImageX itself and 7zFM are happy, and the user is, too: you're right about the fact MS forces the user to name each image when capturing; but they also allow appending and updating freely, without caring about the image name... in fact, name isn't a mandatory field in XML).

    Also, I always think that a program is better coded when it lets more choices and more freedom to the user.

     
  • synchronicity

    synchronicity - 2013-05-21

    I actually just implemented another possible solution: when appending an image where the user did not specify a name, a unique name will be generated if there is a conflict. For example, "dir" would automatically become "dir (1)" when appending a directory named "dir" to a WIM that already contains an image named "dir". This is only when the user does not specify a name, since if the user actually bothers to specify a name then clearly they might want to access the WIM image by that name, which would not be possible if there is a conflict.

    But wimlib still will not let you create an unnamed image even if you explicitly specify an empty string, which as you've brought up is arguably not the ideal behavior. So I might change this as well.

    I guess the final solution would depend on whether it would be preferable for image names to default to the name of the source directory or to unnamed. I thought that the default names would be convenient compared to remembering image indices, which is why I implemented it in the first place, but maybe not--- perhaps it should be assumed the user does not care about the name at all if they do not specify one.

     
  • maxpat78

    maxpat78 - 2013-05-21

    You could simply name it according to its numeric index: surely, it is a unique name...

     
  • synchronicity

    synchronicity - 2013-05-21

    Unfortunately that's not guaranteed. I don't see any reason why I couldn't name the index 1 image "2", in which case there would be a name conflict if the name of the index 2 image defaulted to "2".

    Of course in such cases the names aren't useful anyway because it will not be clear whether the image name or index is being used (wimlib-imagex, for its part, will always interpret a purely numeric string as an image index rather than an image name).

     
  • synchronicity

    synchronicity - 2013-05-21

    I replaced the wimlib-1.4.1-windows-i686-BETA-bin.zip with the latest version. This has:

    • Fix for the volume features detection code.
    • Progress messages after each delete/rename operation as you had suggested above.
    • Appended default image names are changed to avoid conflict as I suggested above.
    • Can create unnamed image by specifying an empty string for the image name.
    • No warning message when opening a WIM with unnamed image(s).
     
  • maxpat78

    maxpat78 - 2013-05-22

    In your latest 1.4.1 (Win): applying an image to . (current) directory does not work, is this the wanted behavior?

     
  • synchronicity

    synchronicity - 2013-05-22

    Probably not. I noticed this as well--- it could be that Windows doesn't allow you to open a directory with write access while it's the working directory of another process. I guess the expected behavior would be just skip extracting the root directory, especially Microsoft's ImageX does not extract the root (confirmed: it failed to extract the root directory as hidden if it has FILE_ATTRIBUTE_HIDDEN).

     
1 2 > >> (Page 1 of 2)

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks