Preventing URLConnection from buffering entire output stream

Posted: 2004-10-15 0:06:00

java-programmer, Preventing URLConnection from buffering entire output stream In my client application, I upload huge files via HTTP.
The URLConnection class seems to automatically buffer
the entire contents, then when the OutputStream associated with
the URLConnection is closed, it computes the length
and prepends a Content-Length header before actually sending
any bytes to the web server.

This results in an OutOfMemoryException when the file is huge.

My application uses Transfer-Encoding: chunked, but
that doesn't help when the entire stream gets buffered anyway.

Is there a way around this without writing my own URLConnection
clone based on Sockets? Calling flush() on the OutputStream
seems to have no effect.

I understand that a workaround would be to use the new features
of Java 1.5, but that's too new to expect end users to have.

Any help would be appreciated.

Thanks!

Mark Riordan

Note: email address in header, email***@***.com, must have
"NoSpam" removed.

Posted: 2004-10-15 1:25:00

java-programmer >> Preventing URLConnection from buffering entire output stream Mark,

I believe, you can do this going at the lower [Client] Socket level.

Create a Client Socket.
Open the input and output streams.
Write the HTTP headers (a GET request I guess).
Read the Inputsream by chunks (you will need to remove the headers
though - anything to the blank line).
Close the socket.

Bruno

Mark Riordan wrote:
> In my client application, I upload huge files via HTTP.
> The URLConnection class seems to automatically buffer
> the entire contents, then when the OutputStream associated with
> the URLConnection is closed, it computes the length
> and prepends a Content-Length header before actually sending
> any bytes to the web server.
>
> This results in an OutOfMemoryException when the file is huge.
>
> My application uses Transfer-Encoding: chunked, but
> that doesn't help when the entire stream gets buffered anyway.
>
> Is there a way around this without writing my own URLConnection
> clone based on Sockets? Calling flush() on the OutputStream
> seems to have no effect.
>
> I understand that a workaround would be to use the new features
> of Java 1.5, but that's too new to expect end users to have.
>
> Any help would be appreciated.
>
> Thanks!
>
> Mark Riordan
>
> Note: email address in header, email***@***.com, must have
> "NoSpam" removed.
>
>

Posted: 2004-10-15 2:23:00

java-programmer >> Preventing URLConnection from buffering entire output stream Bruno,

It is looking as if that's what I will have to do.
Too bad. I've done a lot of web searching, and I'm surprised
that this issue has not come up more frequently for Java programmers.

(In my case, it's a POST and the data is being sent
from my client rather than received, but it's the same idea.)

Thanks.

Mark

"Bruno Grieder" <email***@***.com> wrote in message
news:ckmbjf$n7l$email***@***.com...
> Mark,
>
> I believe, you can do this going at the lower [Client] Socket level.
>
> Create a Client Socket.
> Open the input and output streams.
> Write the HTTP headers (a GET request I guess).
> Read the Inputsream by chunks (you will need to remove the headers
> though - anything to the blank line).
> Close the socket.
>
> Bruno
>
> Mark Riordan wrote:
> > In my client application, I upload huge files via HTTP.
> > The URLConnection class seems to automatically buffer
> > the entire contents, then when the OutputStream associated with
> > the URLConnection is closed, it computes the length
> > and prepends a Content-Length header before actually sending
> > any bytes to the web server.
> >
> > This results in an OutOfMemoryException when the file is huge.
> >
> > My application uses Transfer-Encoding: chunked, but
> > that doesn't help when the entire stream gets buffered anyway.
> >
> > Is there a way around this without writing my own URLConnection
> > clone based on Sockets? Calling flush() on the OutputStream
> > seems to have no effect.
> >
> > I understand that a workaround would be to use the new features
> > of Java 1.5, but that's too new to expect end users to have.
> >
> > Any help would be appreciated.
> >
> > Thanks!
> >
> > Mark Riordan
> >
> > Note: email address in header, email***@***.com, must have
> > "NoSpam" removed.
> >
> >

Posted: 2004-10-15 4:54:00

java-programmer >> Preventing URLConnection from buffering entire output stream
"Mark Riordan" <email***@***.com> wrote in message
news:email***@***.com...
> Bruno,
>
> It is looking as if that's what I will have to do.
> Too bad. I've done a lot of web searching, and I'm surprised
> that this issue has not come up more frequently for Java programmers.
>
> (In my case, it's a POST and the data is being sent
> from my client rather than received, but it's the same idea.)
>
> Thanks.
>
> Mark
>
> "Bruno Grieder" <email***@***.com> wrote in message
> news:ckmbjf$n7l$email***@***.com...
> > Mark,
> >
> > I believe, you can do this going at the lower [Client] Socket level.
> >
> > Create a Client Socket.
> > Open the input and output streams.
> > Write the HTTP headers (a GET request I guess).
> > Read the Inputsream by chunks (you will need to remove the headers
> > though - anything to the blank line).
> > Close the socket.
> >
> > Bruno
> >
> > Mark Riordan wrote:
> > > In my client application, I upload huge files via HTTP.
> > > The URLConnection class seems to automatically buffer
> > > the entire contents, then when the OutputStream associated with
> > > the URLConnection is closed, it computes the length
> > > and prepends a Content-Length header before actually sending
> > > any bytes to the web server.
> > >
> > > This results in an OutOfMemoryException when the file is huge.
> > >
> > > My application uses Transfer-Encoding: chunked, but
> > > that doesn't help when the entire stream gets buffered anyway.
> > >
> > > Is there a way around this without writing my own URLConnection
> > > clone based on Sockets? Calling flush() on the OutputStream
> > > seems to have no effect.
> > >
> > > I understand that a workaround would be to use the new features
> > > of Java 1.5, but that's too new to expect end users to have.
> > >
> > > Any help would be appreciated.
> > >
> > > Thanks!
> > >
> > > Mark Riordan
> > >
> > > Note: email address in header, email***@***.com, must have
> > > "NoSpam" removed.
> > >
> > >
>
>

Excuse my naivet? but why does a POST request need to
send the size?

Posted: 2004-10-15 5:07:00

java-programmer >> Preventing URLConnection from buffering entire output stream "Ann" <email***@***.com> wrote in message news:hHBbd.241452>
> Excuse my naivet? but why does a POST request need to
> send the size?

Technically, it doesn't.
Which is why I'm irritated that Sun's HttpURLConnect apparently
is hard-coded to always send Content-Length at the beginning.
(I say "apparently"; I've love for someone to prove me wrong.)

Now, the web server needs to have some way of figuring out
where the POST data end. But there are ways around this:
the headers "Transfer-Encoding: chunked", and "Connection: close".

However, neither of these techniques seems to affect HttpURLConnection's
unfortunate insistence on sending Content-Length anyway.

Mark R remove NoSpam from email addr

Posted: 2004-10-15 5:15:00

java-programmer >> Preventing URLConnection from buffering entire output stream Mark Riordan wrote:

> Is there a way around this without writing my own URLConnection
> clone based on Sockets? Calling flush() on the OutputStream
> seems to have no effect.
>
> I understand that a workaround would be to use the new features
> of Java 1.5, but that's too new to expect end users to have.
>
> Any help would be appreciated.

I haven't investigated in this context, but I've used the jakarta
project's HTTPClient Library before, and found it much more flexible
than HttpURLConnection. Have you taken a look at it?

Posted: 2004-10-16 2:01:00

java-programmer >> Preventing URLConnection from buffering entire output stream Thanks for the tip. I found some other HTTP libraries
that looked good, but their licensing terms were not as
generous as that of Jakarta.

Mark R

"Carl Howells" <email***@***.com> wrote in message
news:email***@***.com...
> I've used the jakarta
> project's HTTPClient Library before, and found it much more flexible
> than HttpURLConnection. Have you taken a look at it?