Hello again v77,
I'm having some issues with some compatibility with Trend Micro. When trying to format the disk after creation, I'm getting a blue screen error causing the server to reboot. The error involves the AWE allocation driver (C:\Windows\System32\Drivers\awealloc.sys). If Trend is disabled the issue still persists. The only one it works is with Trend completely uninstalled. We've added an exception to the .sys file for Trend with no luck. I was wondering if you had any thoughts regarding additional testing or anything we could try. I don't have a screenshot of the actual blue screen event as I'm on an RDP connection (not the console), but could look to get one if needed.
The drive creates fine, but the formatting is where the issue occurs. There are no event errors posted in Windows, making it tough to narrow down.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi, thanks for the bug report. What is the exact Trend Micro product and version? Would it be possible for you to send me a minidump from the crash? They are usually saved in C:\Windows\minidump. Then I could analyze what was going on in this driver when it crashed.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks. The crash in that dump happened in imdisk.sys and in a worker thread, so it seems that awealloc was not involved. From what I can see imdisk.sys was sent a very strange request that was not handled correctly. I'll try to figure out when/why/how such requests might be sent and how to handle them properly, or if there is something that Trend Micro should take a look at.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I tried setting the Allocate Memory Dynamically with the same results. By default, in the advanced tab, the key Use AWE physical memory is unchecked by default, I assume that is what you mean by disabling the use of AWE? Whether checked or not, they seem to cause a crash still when the formatting process kicks off. I did attach the dump file for Olof.
Thanks again,
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
When "Allocate Memory Dynamically" is checked, the memory allocation is done in user mode by RamDyn.exe, even with AWE. So in this case, it is impossible to have a crash with the AWEAlloc driver (awealloc.sys).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It's possible that the crash is related to something else, I'll try to get console access to see if the bluescreen indicates anything else. However the dump files appear to both reference imdisk.sys, even though the original blue screen does show awealloc.sys. Both dump files with Dynamic and without look pretty similar. I'm working to put an exception to imdisk.sys in Trend as well for testing.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you still have a minidump from the crash that showed awealloc.sys? It would be a bit easier to follow since there is no worker thread involved in the drivers in that case. Everything is handled in a direct call stack down to awealloc.sys within the same thread.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Attached is a dump from when I check the box "Use AWE physical memory". The first dump file I sent didn't have the AWE physical memory checked and it showed awealloc.sys in the blue screen error, so that's strange. But this dump file did specifically use that setting and it does show in the dump file awealloc.sys
So, one of the crashes happened in in awealloc.sys and the other in imdisk.sys, but they are very similar. The only difference is that the use cases cause the I/O buffers to be accessed at different places in the code. In both cases the drivers are sent a read request with an obviously invalid memory descriptor for the I/O buffer (invalid Irp->MdlAddress). This seems to be caused by some manipulation of the request that is done by a driver called tmevtmgr.sys, which is apparently a driver called Trend Micro Event Management.
I cannot investigate this much further because I cannot get that driver to load on a test machine without activation of Trend Micro software. I would recommend that you file a bug report to Trend Micro with this information. They are of course welcome to contact me for more details if needed.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
V77,
Is there a full list of exceptions we could apply to see if Trend would ignore IMDisk? Exceptions for imdisk.sys and awealloc.sys alone don't seem to be enough. I'm not sure if Trend is still just involving itself with the exceptions set or if there are other exceptions that are needed to keep it from getting involved.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If there is something to ignore, it would rather be the volumes created by ImDisk. But of course, this means to ignore all their content.
In my opinion, it's meaningless to ignore imdisk.sys and awealloc.sys. They are simple files on the hard drive, like any other.
That said, this may also not work, because what improperly access the volume is an "Event Manager" from TM. So it depends on how it is linked to the analyze tools.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello again v77,
I'm having some issues with some compatibility with Trend Micro. When trying to format the disk after creation, I'm getting a blue screen error causing the server to reboot. The error involves the AWE allocation driver (C:\Windows\System32\Drivers\awealloc.sys). If Trend is disabled the issue still persists. The only one it works is with Trend completely uninstalled. We've added an exception to the .sys file for Trend with no luck. I was wondering if you had any thoughts regarding additional testing or anything we could try. I don't have a screenshot of the actual blue screen event as I'm on an RDP connection (not the console), but could look to get one if needed.
The drive creates fine, but the formatting is where the issue occurs. There are no event errors posted in Windows, making it tough to narrow down.
Hi, thanks for the bug report. What is the exact Trend Micro product and version? Would it be possible for you to send me a minidump from the crash? They are usually saved in C:\Windows\minidump. Then I could analyze what was going on in this driver when it crashed.
Waiting on the version of the Trend Micro Deep Security. In the meantime I have attached the dump file.
Edit: Deep Security Agent ver 12.0.1090
Last edit: jcrnc 2020-06-19
Thanks. The crash in that dump happened in imdisk.sys and in a worker thread, so it seems that awealloc was not involved. From what I can see imdisk.sys was sent a very strange request that was not handled correctly. I'll try to figure out when/why/how such requests might be sent and how to handle them properly, or if there is something that Trend Micro should take a look at.
The driver part is for Olof, I reported the issue.
For now, as a workaround, you can either use the dynamic memory allocation feature, or disable the use of AWE.
Hi,
I tried setting the Allocate Memory Dynamically with the same results. By default, in the advanced tab, the key Use AWE physical memory is unchecked by default, I assume that is what you mean by disabling the use of AWE? Whether checked or not, they seem to cause a crash still when the formatting process kicks off. I did attach the dump file for Olof.
Thanks again,
When "Allocate Memory Dynamically" is checked, the memory allocation is done in user mode by RamDyn.exe, even with AWE. So in this case, it is impossible to have a crash with the AWEAlloc driver (awealloc.sys).
It's possible that the crash is related to something else, I'll try to get console access to see if the bluescreen indicates anything else. However the dump files appear to both reference imdisk.sys, even though the original blue screen does show awealloc.sys. Both dump files with Dynamic and without look pretty similar. I'm working to put an exception to imdisk.sys in Trend as well for testing.
Do you still have a minidump from the crash that showed awealloc.sys? It would be a bit easier to follow since there is no worker thread involved in the drivers in that case. Everything is handled in a direct call stack down to awealloc.sys within the same thread.
Hi,
Attached is a dump from when I check the box "Use AWE physical memory". The first dump file I sent didn't have the AWE physical memory checked and it showed awealloc.sys in the blue screen error, so that's strange. But this dump file did specifically use that setting and it does show in the dump file awealloc.sys
Thanks! I'll analyze it soon!
So, one of the crashes happened in in awealloc.sys and the other in imdisk.sys, but they are very similar. The only difference is that the use cases cause the I/O buffers to be accessed at different places in the code. In both cases the drivers are sent a read request with an obviously invalid memory descriptor for the I/O buffer (invalid Irp->MdlAddress). This seems to be caused by some manipulation of the request that is done by a driver called tmevtmgr.sys, which is apparently a driver called Trend Micro Event Management.
I cannot investigate this much further because I cannot get that driver to load on a test machine without activation of Trend Micro software. I would recommend that you file a bug report to Trend Micro with this information. They are of course welcome to contact me for more details if needed.
Olof, Thank you so much for your quick responses and investigation. I will be in touch if we get any further.
V77,
Is there a full list of exceptions we could apply to see if Trend would ignore IMDisk? Exceptions for imdisk.sys and awealloc.sys alone don't seem to be enough. I'm not sure if Trend is still just involving itself with the exceptions set or if there are other exceptions that are needed to keep it from getting involved.
If there is something to ignore, it would rather be the volumes created by ImDisk. But of course, this means to ignore all their content.
In my opinion, it's meaningless to ignore imdisk.sys and awealloc.sys. They are simple files on the hard drive, like any other.
That said, this may also not work, because what improperly access the volume is an "Event Manager" from TM. So it depends on how it is linked to the analyze tools.
Can confirm I am experiencing this with Trend Micro Deep Security 20.0.0.3964