I want to make you aware of the new SURF file standard, that is currently not supported by Gwyddion, as well as an issue for files containing image data, such as intensity and rgb data.
In the source code file surffile.c, you check if the file is longer than expected based on the contents of the header. If it is longer, you simply move the filepointer by the "unpexpected" size difference and then read the data at that location.
if (expected_size != size) {
gwy_debug("Size mismatch!");
if (size > expected_size)
add = size - expected_size; /*TODO correct this !*/
else {
err_SIZE_MISMATCH(error, expected_size, size, TRUE);
gwy_file_abandon_contents(buffer, size, NULL);
return NULL;
}
}
sanitise_real_size(&surffile.dx, "x step");
sanitise_real_size(&surffile.dy, "y step");
p = buffer + SURF_HEADER_SIZE + add;
However, this solution mistakenly reads the wrong data as height data. The surf file format can contain more than one sur object. In the header, normally the number of objects is encoded in bytes 14-15. However, this number will only be >1 for multilayer or series studiable types. For rgb + intensity + surface studiables, the number of objects encoded is still 1, even though a total of 5 sur objects are saved in the file. Each object contains its own header again. In this case, the first object is the height data (or surface), followed by three objects for R, G, B channels and a fifth object for the intensity image. What Gwyddion currently does, is mistakenly read the data from the fith object (the intensity) and represents it as height data using the z-Dimensions taken from the first header instead of the fifth header.
The second aspect that Gwyddion does currently not support is the compressed SURF file format, which has the same file extension but starts with the file magic "DSCOMPRESSED".
I can share the offical file standard specification with you, but I'd rather do it by direct email, since I'm not sure that I'm allowed to share it publically. I can then also share some test files.
Best regards,
Frederic
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Please send it to the normal Gwyddion address yeti@gwyddion.net. I have not received anything at any of my university e-mails so far.
When people try to send e-mails to my institutional address they invariably send it to one of my many namesakes (and vice versa). We regularly get review requests, project documents, student questions, etc. intended for some other David Nečas, sometimes even for some working at a different university.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
By the way, there are two other issues I found with the Gwyddion implementation when it comes to exporting to SUR file format.
Gwyddion pads all string fields in the header with null bytes instead of space characters, even though whitespace characters should be used according to the format specification.
Gwyddion sets the studiable type to a hardcoded value of 1 (see line 633 in surffile.c), which corresponds to a profile studiable. However, for surface studiables, the type should be 2 according to the specification (at least if not additional image layers are saved). This can lead some software to be unable to correctly read the Gwyddion exported sur files, even though they are otherwise correct.
Last edit: Frederic Schell 2024-04-22
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As of r26289 all the problems you mentioned should be fixed – except for compression support, which I still have to implement.
I do not quite get how multiple things in one file work. Using the information that every bit of data has its own header, I currently employ the simple strategy of ‘continue reading while the rest of the file looks like more data’. It seems to work and reads 5 images from test.sur, but the relation between the number of objects and number of things in the file is lost on me…
Also, I can see there are now a bunch of additional data types we could import because Gwyddion can work with such data. Like force volume data, various series, etc. But since they are generally more complex than simple images, I will not attempt implementation based on guessing. If someone (you or someone else) is interested in support for additional data types, ask – and provide some examples.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The documentation states that the number of objects encoded in the header is only > 1 for multilayer or series studiables. I assume that multilayer studiables simply contain multiple surfaces and series studiables contain multiple profiles. However, I have no example files for these.
Otherwise, the documentation says, the number of objects in the header is always set to 1 for these additional types of studiables:
The following objects contain more than 1 layer but are considered as a single object, so the value must be set to 1:
Intensity image
RGB image
Intensity surface (deprecated)
RGB surface (deprecated)
Force curve
RGB intensity surface (Surface + Image)
Some of these are deprecated so I wouldn't consider them at the moment. What remains are Intensity image, RGB image, Force curve and RGB intensity surface. I personally have no experience with force curves, but for the other it is pretty straight forward.
If you read the first header in the file and the object count is set to 1, you read one SUR object (consisting of header, which you already read, and data). But if the type of studiable in the first header corresponds to one of these above mentioned objects, you read a fixed additional number of objects.
For RGB image, the total number of layers should be three. So you read two more. For Force curve, I don't know. For RGB intensity surface, you read 4 more additional objects. The first one you read was the height data, then follow the three RGB layers as objects and then the intensity layer. Basically, just check for the type of studiable if nobjects = 1 and manually read another fixed number of objects, the number of which are determined by the studiable type.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the explanation, even though it seems that if nobjects = 1 reading until there is more data should then work just fine. The import module then does not actually have to know how many to read in each case.
For series/multilayers I guess each item may not have a separate header because there are things like dimensions, spacing or offsets of the corresponding axes. That would change the representation but I probably have to wait until someone comes up with some such file.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear David,
I want to make you aware of the new SURF file standard, that is currently not supported by Gwyddion, as well as an issue for files containing image data, such as intensity and rgb data.
In the source code file
surffile.c, you check if the file is longer than expected based on the contents of the header. If it is longer, you simply move the filepointer by the "unpexpected" size difference and then read the data at that location.However, this solution mistakenly reads the wrong data as height data. The surf file format can contain more than one sur object. In the header, normally the number of objects is encoded in bytes 14-15. However, this number will only be >1 for multilayer or series studiable types. For rgb + intensity + surface studiables, the number of objects encoded is still 1, even though a total of 5 sur objects are saved in the file. Each object contains its own header again. In this case, the first object is the height data (or surface), followed by three objects for R, G, B channels and a fifth object for the intensity image. What Gwyddion currently does, is mistakenly read the data from the fith object (the intensity) and represents it as height data using the z-Dimensions taken from the first header instead of the fifth header.
The second aspect that Gwyddion does currently not support is the compressed SURF file format, which has the same file extension but starts with the file magic "DSCOMPRESSED".
I can share the offical file standard specification with you, but I'd rather do it by direct email, since I'm not sure that I'm allowed to share it publically. I can then also share some test files.
Best regards,
Frederic
Please send the specs and files to me personally. The description of the changes is a bit convoluted.
I have sent you an email to your institutional email address with all the necessary info.
Please send it to the normal Gwyddion address yeti@gwyddion.net. I have not received anything at any of my university e-mails so far.
When people try to send e-mails to my institutional address they invariably send it to one of my many namesakes (and vice versa). We regularly get review requests, project documents, student questions, etc. intended for some other David Nečas, sometimes even for some working at a different university.
My bad, hope it reached you now.
Now I've got it, thanks. I hope to look into it soon.
By the way, there are two other issues I found with the Gwyddion implementation when it comes to exporting to SUR file format.
Last edit: Frederic Schell 2024-04-22
As of r26289 all the problems you mentioned should be fixed – except for compression support, which I still have to implement.
I do not quite get how multiple things in one file work. Using the information that every bit of data has its own header, I currently employ the simple strategy of ‘continue reading while the rest of the file looks like more data’. It seems to work and reads 5 images from
test.sur, but the relation between the number of objects and number of things in the file is lost on me…Also, I can see there are now a bunch of additional data types we could import because Gwyddion can work with such data. Like force volume data, various series, etc. But since they are generally more complex than simple images, I will not attempt implementation based on guessing. If someone (you or someone else) is interested in support for additional data types, ask – and provide some examples.
The documentation states that the number of objects encoded in the header is only > 1 for multilayer or series studiables. I assume that multilayer studiables simply contain multiple surfaces and series studiables contain multiple profiles. However, I have no example files for these.
Otherwise, the documentation says, the number of objects in the header is always set to 1 for these additional types of studiables:
Some of these are deprecated so I wouldn't consider them at the moment. What remains are Intensity image, RGB image, Force curve and RGB intensity surface. I personally have no experience with force curves, but for the other it is pretty straight forward.
If you read the first header in the file and the object count is set to 1, you read one SUR object (consisting of header, which you already read, and data). But if the type of studiable in the first header corresponds to one of these above mentioned objects, you read a fixed additional number of objects.
For RGB image, the total number of layers should be three. So you read two more. For Force curve, I don't know. For RGB intensity surface, you read 4 more additional objects. The first one you read was the height data, then follow the three RGB layers as objects and then the intensity layer. Basically, just check for the type of studiable if nobjects = 1 and manually read another fixed number of objects, the number of which are determined by the studiable type.
Thanks for the explanation, even though it seems that if nobjects = 1 reading until there is more data should then work just fine. The import module then does not actually have to know how many to read in each case.
For series/multilayers I guess each item may not have a separate header because there are things like dimensions, spacing or offsets of the corresponding axes. That would change the representation but I probably have to wait until someone comes up with some such file.
I implemented also compression support (r26291). Please try tomorrow's or later development snapshot.