Hi Thomas,

See if adding "-server" helps your import any.

Here's one of the JAVA_OPTS that was floating around the mailing lists for some time.

JAVA_OPTS="-server -Xms256m -Xmx4g -XX:MaxPermSize=256m"

Your 26GB memory limit should be quite a lot, I'm hoping that the machine itself has more than that amount, so that Java doesn't exhaust the whole system.


Also, there's no inherent limit to how many or how large of items can be ingested through this interface, just a matter of smoothing it out as it scales. Typical imports are smaller though, and don't hit issues with some of the potential memory leaks that you've unfortunately likely run across.

Also, I don't remember the exact version of DSpace, but somewhere between 1.6 and 3, the indexing process for batch imports decided it was more efficient to not index each item as it was ingested, but rather do the bulk index at the very end of the import.




________________
Peter Dietz
Longsight
www.longsight.com
peter@longsight.com
p: 740-599-5005 x809


On Thu, Jul 24, 2014 at 10:17 AM, Tim Donohue <tdonohue@duraspace.org> wrote:
Hi Thomas,

Looking more closely at DS-1205, I now realize that it won't inherently
"fix" any existing "OutofMemory problems" itself.  All that DS-1205 does
is add the *option* to use a Read-Only Context (which avoids all
caching)...however that option is disabled by default in the current
codebase (except in a few very specific areas).

DS-930 may help in this scenario, and there have been further bug fixes
to the DSPace AIP process in DSpace 1.8, 3 and 4 -- though none other
jump out as potentially memory related right now.

The other option may be to try and do smaller AIP import sets. For
example, you could create the collection and import just 15,000 objects.
Make sure that works fine, then import the next 15,000, etc.

- Tim


On 7/23/2014 12:13 PM, Avino, Thomas W. (LARC-B702)[LITES] wrote:
> I am trying to run Dspace AIP import on a large collection (155,000)
> items.  I am getting “OutOfMemory problems” after running the import for
> more than 20 hours.
>
> I have set my JAVA_OPTS to 26GB.
>
> I saw 2 different bugs related to this
> https://jira.duraspace.org/browse/DS-930 and
> https://jira.duraspace.org/browse/DS-1205.
>
> There is a patch for DS-930, but I could not find a fix for DS-1205.
>
> We are currently running 1.7.1.
>
> Any help would be appreciated.
>
> Thank you,
>
> Thomas W. Avino
>
> http://t3.gstatic.com/images?q=tbn:ANd9GcRnrGOajbLG51OH18dELWjFdU65XUjF7Rb1nhkW034bbGRqm9BoQU4IlnV-**
>
> *NASA Langley Research Center*
>
> Bldg 1194 Room 302B
>
> Mail Stop 185
>
> Hampton, VA 23681-2199
>
> Phone: (757) 864-8495
>
> Fax: (757) 864-6649
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
>
>
>
> _______________________________________________
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette