Patrick,
the tools loads the data and map into memory. with a 32bit jvm you can
only allocate about 1.3-1.5GB of memory independent of what is in your
machine. with a 64bit jvm more is possible.
the code could be changed to not load the data into memory to deal with
larger files.
the standard SOM algorithms we implemented to not accept a distance
matrix as an input but repeatedly use the data to train the map. I know
there are distance matrix algorithms but they are not in our tool. not
that distance matrix methods have a complexity of O(n^2) while the
standard SOM algorithms have O(n). Of course for particular datasets
either one could be faster, in particular for extremely high
dimensionality as you have described.
yes you can save a png of the maps.
hope this helps
fabian
Patrick De Maziere wrote:
> Hello,
>
> I want to use the ESOM tools starting from a distance matrix since I question
> whether the tools can handle > 7000 vectors with a dimensionality > 800000. I
> used the Euclidean distance metric to construct that distance matrix but I
> now wonder what the next steps are to create a U-matrix map and if I can run
> everything from batch (another program is dealing with these huge datasets
> and with the visualisation of any result). So, I like also to know whether I
> can "export" the U-map to a PNG or other graphic format.
>
> (I tried with a smaller data set (100 vectors, dimension +/- 50000, but even
> then he sometimes fails due to memory problems: out of memory, and setting
> the memory parameter in the script to more than 3GB, e.g., Xmx4096m resulted
> in an error (of the program?) that he could not handle such sizes ....)
>
> Many thanks in advance and best regards,
> Patrick
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
> _______________________________________________
> Databionic-ESOM-User mailing list
> Dat...@li...
> https://lists.sourceforge.net/lists/listinfo/databionic-esom-user
>
>
>
|
