Dynamically expand to hard drive if no more free RAM available
Status: Inactive
Brought to you by:
w77
Menu
▾
▴
Hi,
I recently had an idea that could make RAM disks more usable for a temp folder. The problem with temp folders is you never know how many space is needed. Let's say you have 16GB RAM, but there is some stupid application that writes 18 GB of temp data. Then this application would crash or whatever. And this possibility prevents people from actually using a RAM disk for temp folders.
So my idea would be the following: Extend the partition, if necessary, to a local hard drive, once a limit is reached. For example you can set up a partition of 100GB and set that at most 10GB can be written to RAM and everything beyond that will be written to the local drive. This baiscally extends the dynamic allocation you already implemented.
What do you think about this idea? Would you say, based on you experice, this is possible to achieve?
Best,
Herbert
Last edit: Herbert 2019-03-03
Th whole idea of Ramdisk is not to use the harddisk! Even there is an option just to use physical ram and not extend it to the page file "HARDDISK" (allow AWE).
If you need Ramdisk to use the harddisk, just uncheck that option (allow AWE) which is disable by default and extend your page file
Last edit: paradis pal 2019-03-03
So in your opinion, you would then just not use the RAM disk for use cases where very seldomly the limit of the RAM is exceeded. (or where no limit can be guaranteed) In my example case I assume that 99% of the data written is in the first 10GB range and only 1% or so exceeds that limit.
By just dismissing this idea you would end up using less RAM disk than with my proposal where I open the concept of RAM disk to whole new use cases that were not possible before. Just think about it.
I have thought about page file earlier, but the problem is that all content is affected by swapping not just the RAM disk so you would potentially slow down the system a lot once that point is reached.
Last edit: Herbert 2019-03-03
It's not my opinion, I told what the idea about Ramdisk.
Not sure if it is usefull to add an option to extend ramdisk to HDD, but you can make symbolic links to get over it if that may help. Consider using mklink.exe /d
Not sure what app uses 18GB temp, but if so and it is in one folder, get more RAM.
I have 64GB Ram with 40 gb Ramdisk!
Hope the developer found it usefull and apply it for you :)
Last edit: paradis pal 2019-03-03
I once had this idea. Don't know if this is useful but it would be rather difficult to implement, with a lot of code to add. This is a complete new memory management to add, except that instead being in RAM, it would be in a file.
Moreover, as said by paradis pal, relying on the page file can produce a similar effect, in a way that can be more efficient.
So, for now, I prefer to keep the dynamic ramdisks as they are.
Thanks v77 for your remarks on this.
A lot of installers for example extract themself into a temporary folder. Some of those can be as large as several gigabytes. Also if you extract zip files, etc. This is just super wastefull usage of limited write cycles of a SSD. If you are not an entitled rich kid like other people who just buy insame amount of RAM, you probably won't have enogh RAM to savely put the temp folder in there (because there is no guranatee about the maximum file size) and this feature will be very useful.
ImDisk Toolkit is currently the closest one can have in terms of an actually useful RAM disk for temp folders. Because of this dynamic allocation feature. And the fact it is open source is even more inrcedible!
Since 1607 (server 2016) Windows avoids the pagefile as much as possible. It has it's own "pagefile in ram"-idea, see PoSH commands enable-MMAgent. That being said, three's no solution to this for a physical machine. For a virtual machine however, I've solved this problem elegantly by having a ramdisk that's bigger than available RAM. Then, when there's ram-shortage, I have a small pagefile (2GB) as buffer, before on the Host Hyper-V Dynamic Memory will kick in and grant the VM more Ram. Thus, that way, it's as dynamic as you can get it without running out of RamDisk Space as defined by the lower limits. Ofcourse you still have an upper limit, that's unavoidable in the end. On a host with alot of VMs this really is a gamechanger, both in terms of "spend" IOps aswell as VM-density, as each VM only uses as much RAM as really needed WHEN it's needed, and releases it again thereafter, first to the VM, then the VM to the Host, enabling many more VMs to be run from the same host.
would love to see the fail safe spillover hard drive too.
want to implement ramdisk but am afriad something may fill it up and crash the whole system. what exactly happens when it fills up, does everything just grind to a halt?
You get an error stating it's unable to write to the RamDisk, which has the effect that it stops everything else indeed, until you click OK, then everything moves along again. No real crash will happen, just a perceived "hang". On a multi-user server, quite annoying ofcourse. They won't see the error, but everything starts "hanging" for them out of nowhere.
So can this be implemented? It is a bit unclear from the comments in this thread whether it is/isn't possible etc.
Possible? Yes (maybe with a bit less work than I thought).
Does it worth it? I still think that no.
If you fear to get data loss by running out of RAM, you should rely on the page file. In most cases, it will provide better overall performance than with this kind of addition.
And if you worry about write cycles of a SSD, you can move the page file to another drive, like you would do with an extension file of ramdisk.
Running out of RAM on the imdisk has happened to me, so I think this is
worthy of consideration.
However, my higher priority is incremental loading and unloading (vs the
current bulk load/unload).
Thx
On Tue, Jun 4, 2019, 9:33 AM v77 w77@users.sourceforge.net wrote:
If you're just concerned about "unnecessary" writes to SSD, consider this solution, a bit cumbersome to setup, but nevertheless works elegantly: Have your dyn ramdisk setup, using diskpart, create a VHDx-file, make sure it's on your HDD/SSD. Next, create a second VHDx on your ramdisk, with parameter parent=<vhdx on="" hdd="">. Attach it and mount it as driveletter or mountpoint. Use mklink / d to make links to it (only need to do this once). Doing this, you create a differential VHDx in ram, where only new writes will go. when you're done, or @ shutdown, use diskpart once more to merge the child VHDx with the parent. You've got a quick "load" this way, since there's nothing to load, and you got a quick "save" as it only has to merge content that was newly written. You don't need to load/save or sync your ramdisk any other way. You will use less ram too, as only newly written content will go there. Since you mount a VHDx, added benefit is that NTFS compression will correctly work, if you (schedule) defrag your mountpoint (which is otherwise not possible as the dyn ramdisk itself is not a real volume). Any disk-functions will work. There's only one downside I encountered. If this is inside a VM and you live backup that VM using VSS, it will fail, as VSS will fail in the presence of a mounted VHDx which has a child on a "not real volume", as the dyn ramdisk is.</vhdx>
Wow, thx!
Think you could turn this whole process into a one-click process in your application?
:)
On 6/4/2019 12:31:31 PM, Wizard wizard17@users.sourceforge.net wrote:
If you're just concerned about "unnecessary" writes to SSD, consider this solution, a bit cumbersome to setup, but nevertheless works elegantly: Have your dyn ramdisk setup, using diskpart, create a VHDx-file, make sure it's on your HDD/SSD. Next, create a second VHDx on your ramdisk, with parameter parent=<vhdx on="" hdd="">. Attach it and mount it as driveletter or mountpoint. Use mklink / d to make links to it (only need to do this once). Doing this, you create a differential VHDx in ram, where only new writes will go. when you're done, or @ shutdown, use diskpart once more to merge the child VHDx with the parent. You've got a quick "load" this way, since there's nothing to load, and you got a quick "save" as it only has to merge content that was newly written. You don't need to load/save or sync your ramdisk any other way. You will use less ram too, as only newly written content will go there. Since you mount a VHDx, added benefit is that NTFS compression will correctly work, if you (schedule) defrag your mountpoint (which is otherwise not possible as the dyn ramdisk itself is not a real volume). Any disk-functions will work. There's only one downside I encountered. If this is inside a VM and you live backup that VM using VSS, it will fail, as VSS will fail in the presence of a mounted VHDx which has a child on a "not real volume", as the dyn ramdisk is.</vhdx>
Dynamically expand to hard drive if no more free RAM available [https://sourceforge.net/p/imdisk-toolkit/discussion/general/thread/2d5d79d2f9/?limit=100#b3cf]
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/imdisk-toolkit/discussion/general/ [https://sourceforge.net/p/imdisk-toolkit/discussion/general/]
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/ [https://sourceforge.net/auth/subscriptions/]
That's not really something I think belongs in this product, as it's totally custom built. I'm simply using 2 powershell scripts, one at startup and one at shutdown for that... I wanted to convey a possible solution to the features raised. It will involve some scripting to get this working to personal preference. Im just using imdisk for it as a complement tool to accomplish this goal.