Re: [Resteasy-users] FileUpload with big files failing withOutOfMemory Error

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

FYI, there's a good chance I'm going to be deprecating Apache Client 4 
in Resteasy 3.0, then removing it in 3.1.  We'll be switching to our own 
custom code.  Particularly because of some of the headers issues I 
discussed before.

Also, we do have a Resteasy client cache that you might want to take a 
look at.

On 6/8/12 11:12 AM, Sandeep Tikoo wrote:
> Hi,
>
> I cannot use URLConnectionClientExecutor because I need to use Apache's
> Http Caching component.
> Not sure if I understood the issue with Apache Client's RequestEntity
> and how that forces having an in-memory Stream but I would appreciate it
> very much if you could have a look at the attached files and let me know
> if I could run into any issues down the line.
> I am at the moment extending ApacheHttpClient4Executor.java with my
> custom class; however, the attached file
> (ApacheHttpClient4Executor_2.1.1.GA_PATCHED.java) presents the fix on
> top of the original file to facilitate easier comparison. The patch.txt
> file is generated using WinMerge.
>
> Thank you,
> Sandeep.
>
>
> -----Original Message-----
> From: Bill Burke [mailto:bb...@re...]
> Sent: Thursday, June 07, 2012 3:23 PM
> To: res...@li...
> Subject: Re: [Resteasy-users] FileUpload with big files failing
> withOutOfMemory Error
>
> Yes please elaborate on your fix.
>
> FYI, the problem I had with Apache Client 3/4 RequestEntity was that you
> couldn't add/modify headers within the RequestEntity implementation.
> MessageBodyWriters are allowed to add/modify headers before they write
> to the output stream.  So, in order to fulfill this requirement on top
> of Apache Client, the entity needs to be buffered.
>
> I don't think you would see this same OOM problem if you used the
> URLConnectionClientExecutor because it doesn't have the same silly
> constraint as AC 3/4.
>
> On 6/7/12 2:54 PM, Sandeep Tikoo wrote:
>> Realized that https://issues.jboss.org/browse/RESTEASY-233 fixed it on
>
>> the server side only and not on the client-side. I have, for now, to
>> keep my project moving forward, applied a fix on top of
>> org.jboss.resteasy.client.core.executors.ApacheHttpClient4Executor
>> using apache common's DefferedFileOutputStream
>>
> (http://commons.apache.org/io/apidocs/org/apache/commons/io/output/Defer
> redFileOutputStream.html).
>>
>>
>> If there is interest, I would be more than happy to share the fix.
>>
>> Thank you,
>>
>> Sandeep.
>>
>> *From:*Sandeep Tikoo [mailto:st...@di...]
>> *Sent:* Tuesday, May 15, 2012 6:31 PM
>> *To:* res...@li...
>> *Subject:* Re: [Resteasy-users] FileUpload with big files failing
>> withOutOfMemory Error
>>
>> Hi,
>>
>> I am using RestEasy on the client side (version 2.2.1.GA) and trying
>> to upload a huge file and I get a OutOfMemory Error:
>>
>> *//*
>>
>> */Caused by: java.lang.OutOfMemoryError: Java heap space/*
>>
>> */at java.util.Arrays.copyOf(Arrays.java:2786)/*
>>
>> */at
>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.HeaderFlushedOutputStre
>> am.write(HeaderFlushedOutputStream.java:56)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.InputStreamProvider.writeTo(Input
>> StreamProvider.java:60)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.InputStreamProvider.writeTo(Input
>> StreamProvider.java:22)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartWriter
>> .writePart(AbstractMultipartWriter.java:62)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartFormDa
>> taWriter.writeParts(AbstractMultipartFormDataWriter.java:32)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartWriter
>> .write(AbstractMultipartWriter.java:33)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.MultipartFormAnnotation
>> Writer.writeTo(MultipartFormAnnotationWriter.java:78)/*
>>
>> */at
>> org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proc
>> eed(MessageBodyWriterContextImpl.java:117)/*
>>
>> */at
>> org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingIntercept
>> or.write(GZIPEncodingInterceptor.java:98)/*
>>
>> */at
>> org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proc
>> eed(MessageBodyWriterContextImpl.java:123)/*
>>
>> */at
>> org.jboss.resteasy.client.ClientRequest.writeRequestBody(ClientRequest
>> .java:472)/*
>>
>> *//*
>>
>> *//*
>>
>> *//*
>>
>> It seems that this issue was addressed a long time ago using mime4j:
>> https://issues.jboss.org/browse/RESTEASY-233
>>
>> I do have mime4j in my classpath but not sure if there is something I
>> need to do differently to make this work?
>>
>> Here is my code:
>>
>> The JAX-RS annotated interface:
>>
>> @Path("/resource")
>>
>> *public**interface*RestClientAPI {
>>
>> @POST
>>
>> @Path(/UPDATE_PRODUCTS_ASYNC_RESOURCE/)
>>
>> //@Consumes(MediaType.APPLICATION_XML)
>> <mailto://@Consumes(MediaType.APPLICATION_XML)>
>>
>> @Consumes(MediaType./MULTIPART_FORM_DATA/)
>>
>> @Produces(MediaType./APPLICATION_XML/)
>>
>> Response upload(@MultipartForm*final*LoaderMultiPartForm loader);
>>
>> }
>>
>> *The MultiPartForm class:*
>>
>> *public**class*LoaderMultiPartForm {
>>
>> @FormParam("loader")
>>
>> @PartType(MediaType./APPLICATION_XML/)
>>
>> *private*Loader loader;
>>
>> @FormParam("bulkProductTypeStream")
>>
>> @PartType(MediaType./APPLICATION_OCTET_STREAM/)
>>
>> *private*InputStream bulkProductTypeStream;
>>
>> *public*Loader getLoader() {
>>
>> *return**this*.loader;
>>
>> }
>>
>> *public**void*setLoader(Loader loader) {
>>
>> *this*.loader= loader;
>>
>> }
>>
>> *public*InputStream getBulkProductTypeStream() {
>>
>> *return**this*.bulkProductTypeStream;
>>
>> }
>>
>> *public**void*setBulkProductTypeStream(InputStream
>> bulkProductTypeStream) {
>>
>> *this*.bulkProductTypeStream= bulkProductTypeStream;
>>
>> }
>>
>> }
>>
>> The Call:
>>
>> RestClientAPI client = factory.getClient(CatalogResource.*class*);
>>
>> Loader loader = *new*Loader();
>>
>> loader.setXXX();
>>
>> LoaderMultiPartForm loaderForm = *new*LoaderMultiPartForm();
>>
>> loaderForm.setLoader(loader);
>>
>> hugeFileUploadStream =
>> *new*BufferedInputStream(*new*FileInputStream(*new*File("C:\\tmp\\huge
>> FileToUpload.txt")));
>>
>> loaderForm.setBulkProductTypeStream(hugeFileUploadStream);
>>
>> client.upload(loaderForm);
>>
>> Appreciate any help.
>>
>> Regards,
>>
>> Sandeep.
>>
>>
>>
>> ----------------------------------------------------------------------
>> --------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
>> Discussions will include endpoint security, mobile security and the
>> latest in malware threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>>
>> _______________________________________________
>> Resteasy-users mailing list
>> Res...@li...
>> https://lists.sourceforge.net/lists/listinfo/resteasy-users
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
>
>
> ------------------------------------------------------------------------
> ------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond.
> Discussions
> will include endpoint security, mobile security and the latest in
> malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Resteasy-users mailing list
> Res...@li...
> https://lists.sourceforge.net/lists/listinfo/resteasy-users

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com