HTTP creep

java-programmer, HTTP creep If you look at the packet headers going back and forth to your HTTP
server, you will discover a HECK of a lot of blather that really is
not doing much.

It is made worse by it all being human readable without abbreviations.

Add this overhead up on all packet headers being transferred each day
and you see a stupendous waste of bandwidth.

HTTP should be put on a diet, with rigid defaults, bitfield headers,
and compressed messages.

The original protocol was designed for maximum flexibility. Now we
have some experience, we can tighten it up.

It matters now that you have tiny handhelds trying to surf the net
over limited bandwidth.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

java-programmer >> HTTP creep
"Roedy Green" <email***@***.com> wrote in message
news:email***@***.com...
> If you look at the packet headers going back and forth to your HTTP
> server, you will discover a HECK of a lot of blather that really is
> not doing much.
>
> It is made worse by it all being human readable without abbreviations.
>
> Add this overhead up on all packet headers being transferred each day
> and you see a stupendous waste of bandwidth.
>
> HTTP should be put on a diet, with rigid defaults, bitfield headers,
> and compressed messages.
>
> The original protocol was designed for maximum flexibility. Now we
> have some experience, we can tighten it up.
>
> It matters now that you have tiny handhelds trying to surf the net
> over limited bandwidth.

Compare total text size to total image size --how much are you saving?

java-programmer >> HTTP creep Roedy Green <email***@***.com> writes:

> HTTP should be put on a diet, with rigid defaults, bitfield headers,
> and compressed messages.

... and SOAP should be turned back into EDIFACT.

Won't happen that either, though.

java-programmer >> HTTP creep Roedy Green wrote:

> If you look at the packet headers going back and forth to your HTTP
> server, you will discover a HECK of a lot of blather that really is
> not doing much.
>
> It is made worse by it all being human readable without abbreviations.
>
> Add this overhead up on all packet headers being transferred each day
> and you see a stupendous waste of bandwidth.
>
> HTTP should be put on a diet, with rigid defaults, bitfield headers,
> and compressed messages.
>
> The original protocol was designed for maximum flexibility. Now we
> have some experience, we can tighten it up.
>
> It matters now that you have tiny handhelds trying to surf the net
> over limited bandwidth.

Most modern Web browsers support gzipped output from Web servers, and most
servers support gzipping output. There is already technology in place to
put HTTP on a diet, but no one uses it: <url:
http://webreference.com/internet/software/servers/http/compression/ />

I can hear your objection already, compressing the output of all your Web
pages everytime they are requested would place a considerable load on the
HTTP server, so I propose some mechanism by which all static content is
pre-gzipped, and then simply served from disk.

For dynamically created content, it would have to be compressed each time,
but then again, your proposed solution would suffer from the same problem,
each dynamically created resource would have to be "encoded" by whatever
means you decide, so there would always be overhead for dynamic content.

Of course, the real problem isn't implementing something on the server,
even something that is resource intensive (after all, we can always throw
more hardware at the problem), the real problem is there are 30+ user
agents in the world, running on millions of computers, and they all expect
RFC2616-compliant headers. Even if servers supported your proposed
functionality tomorrow, we would be supporting HTTP/1.1 for many, many,
many years to come.

Not to mention, what do you propose I do with a Perl script that does:

print "Content-Type: text/html\n";
print "X-my-custom-header: value;value;value\n";
print "Content-Length: 0\n";
print "\n";

?

java-programmer >> HTTP creep
"Grant Wagner" <email***@***.com> wrote in message
news:email***@***.com...
> Roedy Green wrote:
>
> > If you look at the packet headers going back and forth to your HTTP
> > server, you will discover a HECK of a lot of blather that really is
> > not doing much.
> >
> > It is made worse by it all being human readable without abbreviations.
> >
> > Add this overhead up on all packet headers being transferred each day
> > and you see a stupendous waste of bandwidth.
> >
> > HTTP should be put on a diet, with rigid defaults, bitfield headers,
> > and compressed messages.
> >
> > The original protocol was designed for maximum flexibility. Now we
> > have some experience, we can tighten it up.
> >
> > It matters now that you have tiny handhelds trying to surf the net
> > over limited bandwidth.
>
> Most modern Web browsers support gzipped output from Web servers, and most
> servers support gzipping output. There is already technology in place to
> put HTTP on a diet, but no one uses it: <url:
> http://webreference.com/internet/software/servers/http/compression/ />
>
> I can hear your objection already, compressing the output of all your Web
> pages everytime they are requested would place a considerable load on the
> HTTP server, so I propose some mechanism by which all static content is
> pre-gzipped, and then simply served from disk.
>
> For dynamically created content, it would have to be compressed each time,
> but then again, your proposed solution would suffer from the same problem,
> each dynamically created resource would have to be "encoded" by whatever
> means you decide, so there would always be overhead for dynamic content.
>
> Of course, the real problem isn't implementing something on the server,
> even something that is resource intensive (after all, we can always throw
> more hardware at the problem), the real problem is there are 30+ user
> agents in the world, running on millions of computers, and they all expect
> RFC2616-compliant headers. Even if servers supported your proposed
> functionality tomorrow, we would be supporting HTTP/1.1 for many, many,
> many years to come.
>
> Not to mention, what do you propose I do with a Perl script that does:
>
> print "Content-Type: text/html\n";
> print "X-my-custom-header: value;value;value\n";
> print "Content-Length: 0\n";
> print "\n";
>
> ?
>
> --
> | Grant Wagner <email***@***.com>
>

Roedy talks about headers. GZIP compression in HTTP compresses the
request/response bodies, not the headers.

It only matters if you use HTTP as a transport protocol for a very dense
application level protocol because body sizes can be relatively small then.
In such cases it is best to use as few headers as possible. Quite soon
bandwith will not be so limited even for handhelds so I do not think HTTP
should be altered. When it was invented people had the same objections for
use in plain computers.

Silvio Bierman

java-programmer >> HTTP creep On 2004-07-05, Grant Wagner <email***@***.com> wrote:
> Roedy Green wrote:
>> If you look at the packet headers going back and forth to your HTTP
>> server, you will discover a HECK of a lot of blather that really is
>> not doing much.
...
> Most modern Web browsers support gzipped output from Web servers, and most
> servers support gzipping output. There is already technology in place to
> put HTTP on a diet, but no one uses it: <url:
> http://webreference.com/internet/software/servers/http/compression/ />

That compresses that page content, not the HTTP headers.

--
--Tim Smith

java-programmer >> HTTP creep On Mon, 05 Jul 2004 21:09:33 GMT, Grant Wagner
<email***@***.com> wrote or quoted :

>I can hear your objection already, compressing the output of all your Web
>pages everytime they are requested would place a considerable load on the
>HTTP server, so I propose some mechanism by which all static content is
>pre-gzipped, and then simply served from disk.

From a pure economics point of view, which is cheaper, adding extra
cpu power to do the compression on adding extra bandwidth?

From the client point of view, compression should always be faster
unless you were really strapped for CPU power. Unzipping is a lot
faster than zipping.

There are other types of compression you could use that would be
faster to compose. see
http://mindprod.com/projects/htmlcompactor.html

You could glue precompressed bits together with other bits compressed
on the fly.

The big problem is the inertia of converting to anything else. That
is why I want to wring the necks of people who released half-assed
inefficient interconnection protocols. You can never fix them later.

XML is another such idiocy.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

java-programmer >> HTTP creep On Mon, 05 Jul 2004 22:32:23 GMT, Tim Smith
<email***@***.com> wrote or quoted :

>That compresses that page content, not the HTTP headers.

The point is not that the headers need to be compressed, but rather
dispensed with or abbreviated. Everyone keeps tacking just one more
field on.

Watch with a packet sniffer. You will be amazed how bloated they have
become. It is a bit like waistline creep.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

java-programmer >> HTTP creep In comp.lang.java.advocacy, Roedy Green
<email***@***.com>
wrote
on Mon, 05 Jul 2004 16:29:42 GMT
<email***@***.com>:
> If you look at the packet headers going back and forth to your HTTP
> server, you will discover a HECK of a lot of blather that really is
> not doing much.
>
> It is made worse by it all being human readable without abbreviations.
>
> Add this overhead up on all packet headers being transferred each day
> and you see a stupendous waste of bandwidth.
>
> HTTP should be put on a diet, with rigid defaults, bitfield headers,
> and compressed messages.
>
> The original protocol was designed for maximum flexibility. Now we
> have some experience, we can tighten it up.
>
> It matters now that you have tiny handhelds trying to surf the net
> over limited bandwidth.
>

The HTTP protocol seems to be a bit of a mess.

[1] HTTP is over TCP. OK, TCP contains state. No problem yet.

[2] HTTP is stateless. Huh? Wait...why not use UDP? Of course
there are issues with UDP routing on the Internet so TCP
is still defensible, but...

[3] Cookies. Yeppers, the browsers got hungry! Now we have
state again.

In its way, ASN.1 is worse (since nobody seems to
understands it!) but there's an awful lot of gunk in HTTP
that's not really needed; the above is but one example.
The Accept: header is another issue; the original spec
of HTML (which HTTP was designed to transport originally)
was that unrecognized tags were simply ignored. However,
I'd have to look regarding unrecognized image formats in
<IMG> tags.

Sigh. At least Java's RMI protocol makes some sense, even if
it is proprietary... :-) Create a stub on the local machine,
which downloads relevant stuff from the remote.

--
#191, email***@***.com
It's still legal to go .sigless.

java-programmer >> HTTP creep Roedy Green wrote:
> If you look at the packet headers going back and forth to your HTTP
> server, you will discover a HECK of a lot of blather that really is
> not doing much.
>
> It is made worse by it all being human readable without abbreviations.

Being human readable without abbreviations is a /good/ thing, and not
something to be sacrificed without good reason.

> Add this overhead up on all packet headers being transferred each day
> and you see a stupendous waste of bandwidth.

"Stupendous waste"? What fraction of bandwidth is used by HTTP headers?
This is /not/ the bottleneck for web apps IME.

> HTTP should be put on a diet, with rigid defaults, bitfield headers,

Yuck! You don't apply the same principles to your source code, so why
should HTTP headers be any different? Ironically, it is probably the
non-abbreviated human-readable format, that enabled you to prematurely argue
for obfuscating it in the name of optimization.

java-programmer >> HTTP creep On Tue, 06 Jul 2004 22:36:14 GMT, "Justin Farley" <email***@***.com>
wrote or quoted :

>"Stupendous waste"? What fraction of bandwidth is used by HTTP headers?
>This is /not/ the bottleneck for web apps IME.

Yes studendous! What do you think the planet is paying each year for
datacommunication services, the total amount in billions? Hint it
makes up about 7% of all consumer spending.

1% of a billion is $10,000,000.00

Heavens! even that's more than I make in a month.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

java-programmer >> HTTP creep Justin Farley wrote:
> Roedy Green wrote:
[snip]
>
>>HTTP should be put on a diet, with rigid defaults, bitfield headers,
>
>
> Yuck! You don't apply the same principles to your source code, so why
> should HTTP headers be any different?

Because HTTP headers are not source code.

Yikes, I just agreed with Roeds. I feel dirty.

java-programmer >> HTTP creep In comp.lang.java.advocacy, Justin Farley
<email***@***.com>
wrote
on Tue, 06 Jul 2004 22:36:14 GMT
<yPFGc.19083826$email***@***.com>:
> Roedy Green wrote:
>> If you look at the packet headers going back and forth to your HTTP
>> server, you will discover a HECK of a lot of blather that really is
>> not doing much.
>>
>> It is made worse by it all being human readable without abbreviations.
>
> Being human readable without abbreviations is a /good/ thing, and not
> something to be sacrificed without good reason.
>
>> Add this overhead up on all packet headers being transferred each day
>> and you see a stupendous waste of bandwidth.
>
> "Stupendous waste"? What fraction of bandwidth is used by HTTP headers?
> This is /not/ the bottleneck for web apps IME.

If the headers get to be a significant fraction of an ethernet frame
(1460 bytes, IIRC), start worrying. :-)

A sniff of a Lynx request generates the following:

GET http://www.cnn.com HTTP/1.0
Host: www.cnn.com
Accept: text/html, text/plain, application/vnd.sum.xml.writer,
application/vnd.sun.xml.writer.global, application/vnd.stardivision.writer,
application/vnd.stardivision.writer-global, application/x-starwriter,
application/vnd.sun.xml.writer.template
Accept: application/msword, application/vnd.sum.xml.calc,
application/vnd.stardivision.calc, application/x-starcalc,
application/vnd.sun.xml.calc.template, application/excel,
application/msexcel, application/vnd.ms-excel, application/x-msexcel,
Accept: application/vnd.sun.xml.impress, application/vnd/stardivision.impress,
application/vnd.stardivision.impress-packed, application/x-starimpress,
application/vnd.sun.xml.impress.template, application/powerpoint,
application/mspowerpoint
Accept: application/vnd.ms-powerpoint, application/x-mspowerpoint,
application/vnd.sun.xml.draw, application/x-stardraw,
application/vnd.sun.xml.draw.template, application/vnd.sun.xml.math
Accept: application/vnd.stardivision.math, application/x-starmath, text/sgml,
*/*;q=0.01
Accept-Encoding: gzip, compress
Accept-Language: en
Pragma: no-cache
Cache-control: no-cache
User-Agent: Lynx/2.8.5rel.1 libwww-FM/2.14.SSL-MM/1.4.1.OpenSSL/0.9.7d
Referer: http://www.cnn.com/

This is the text (if I've done this correctly!) from byte offset 0x42
to 0x55c -- 1,307 bytes. I've wrapped it to keep it readable.

And this is *before* one adds in the 2047-byte URL requirement.
Ye gods. I'm hoping there's a tweak in /etc/lynx/lynx.cfg to
put this on a diet. I don't know where it's getting all this
from offhand.

(There is one saving grace: Lynx doesn't download pictures unless
specifically asked.)

wget is much more reasonable:

GET http://www.cnn.com/ HTTP/1.0
User-Agent: Wget/1.9
Host: www.cnn.com
Accept: */*

though part of that is because wget is a lot stupider; all it
needs to do is put the results somewhere.

I'll have to look to see what Mozzie does; I'm remotely logged
in at the moment and don't have SSH/X forwarding currently.

>
>> HTTP should be put on a diet, with rigid defaults, bitfield headers,
>
> Yuck! You don't apply the same principles to your source code, so why
> should HTTP headers be any different? Ironically, it is probably the
> non-abbreviated human-readable format, that enabled you to prematurely
> argue for obfuscating it in the name of optimization.

Maybe, but if 10M browsers are out there each generating 1K
per request, that's an extra 100Gb of bandwidth. If one
browses an average of a page a minute, that's 1.6 Gb/s
just throwing headers around.

Of course, compared to the viruses that's probably pretty paltry. :-)

>
> --
> Justin
>
>

--
#191, email***@***.com
It's still legal to go .sigless.

java-programmer >> HTTP creep On Wed, 07 Jul 2004 03:41:19 GMT, The Ghost In The Machine
<email***@***.com> wrote or quoted :

>A sniff of a Lynx request generates the following:

and this does not even count the headers inside the HTML itself that
manages for example to use about 80 bytes to give 4 bits of data --
which of 16 "standards" are being used.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.

java-programmer >> HTTP creep
"The Ghost In The Machine" <email***@***.com> wrote in
message news:email***@***.com...
> In comp.lang.java.advocacy, Justin Farley
> <email***@***.com>
> wrote
> on Tue, 06 Jul 2004 22:36:14 GMT
> <yPFGc.19083826$email***@***.com>:
> > Roedy Green wrote:
> >> If you look at the packet headers going back and forth to your HTTP
> >> server, you will discover a HECK of a lot of blather that really is
> >> not doing much.
> >>
> >> It is made worse by it all being human readable without abbreviations.
> >
> > Being human readable without abbreviations is a /good/ thing, and not
> > something to be sacrificed without good reason.
> >
> >> Add this overhead up on all packet headers being transferred each day
> >> and you see a stupendous waste of bandwidth.
> >
> > "Stupendous waste"? What fraction of bandwidth is used by HTTP headers?
> > This is /not/ the bottleneck for web apps IME.
>
> If the headers get to be a significant fraction of an ethernet frame
> (1460 bytes, IIRC), start worrying. :-)
>
> A sniff of a Lynx request generates the following:
>
> GET http://www.cnn.com HTTP/1.0
> Host: www.cnn.com
> Accept: text/html, text/plain, application/vnd.sum.xml.writer,
> application/vnd.sun.xml.writer.global,
application/vnd.stardivision.writer,
> application/vnd.stardivision.writer-global, application/x-starwriter,
> application/vnd.sun.xml.writer.template
> Accept: application/msword, application/vnd.sum.xml.calc,
> application/vnd.stardivision.calc, application/x-starcalc,
> application/vnd.sun.xml.calc.template, application/excel,
> application/msexcel, application/vnd.ms-excel, application/x-msexcel,
> Accept: application/vnd.sun.xml.impress,
application/vnd/stardivision.impress,
> application/vnd.stardivision.impress-packed, application/x-starimpress,
> application/vnd.sun.xml.impress.template, application/powerpoint,
> application/mspowerpoint
> Accept: application/vnd.ms-powerpoint, application/x-mspowerpoint,
> application/vnd.sun.xml.draw, application/x-stardraw,
> application/vnd.sun.xml.draw.template, application/vnd.sun.xml.math
> Accept: application/vnd.stardivision.math, application/x-starmath,
text/sgml,
> */*;q=0.01
> Accept-Encoding: gzip, compress
> Accept-Language: en
> Pragma: no-cache
> Cache-control: no-cache
> User-Agent: Lynx/2.8.5rel.1 libwww-FM/2.14.SSL-MM/1.4.1.OpenSSL/0.9.7d
> Referer: http://www.cnn.com/
>
> This is the text (if I've done this correctly!) from byte offset 0x42
> to 0x55c -- 1,307 bytes. I've wrapped it to keep it readable.
>
> And this is *before* one adds in the 2047-byte URL requirement.
> Ye gods. I'm hoping there's a tweak in /etc/lynx/lynx.cfg to
> put this on a diet. I don't know where it's getting all this
> from offhand.
>
> (There is one saving grace: Lynx doesn't download pictures unless
> specifically asked.)
>
> wget is much more reasonable:
>
> GET http://www.cnn.com/ HTTP/1.0
> User-Agent: Wget/1.9
> Host: www.cnn.com
> Accept: */*
>
> though part of that is because wget is a lot stupider; all it
> needs to do is put the results somewhere.
>
> I'll have to look to see what Mozzie does; I'm remotely logged
> in at the moment and don't have SSH/X forwarding currently.
>
> >
> >> HTTP should be put on a diet, with rigid defaults, bitfield headers,
> >
> > Yuck! You don't apply the same principles to your source code, so why
> > should HTTP headers be any different? Ironically, it is probably the
> > non-abbreviated human-readable format, that enabled you to prematurely
> > argue for obfuscating it in the name of optimization.
>
> Maybe, but if 10M browsers are out there each generating 1K
> per request, that's an extra 100Gb of bandwidth. If one
> browses an average of a page a minute, that's 1.6 Gb/s
> just throwing headers around.
>
> Of course, compared to the viruses that's probably pretty paltry. :-)
>
> >
> > --
> > Justin
> >
> >
>
>
> --
> #191, email***@***.com
> It's still legal to go .sigless.

Who on earth created lynx? This looks totally stupid. The accept header is
at best moderately usefull but never when used as it is here.

Although I would not like to see HTTP becoming less readable I would applaud
any initiative to set some standards on when to set which headers, which
should be done with care.

Silvio Bierman

java-programmer >> HTTP creep HTTP should be done using XML - i.e. that's probably how it would
be done if the protocol were reinvented today ;-)
--
__________
|im |yler http://timtyler.org/ email***@***.com Remove lock to reply.

java-programmer >> HTTP creep Tim Tyler <email***@***.com> writes:

> HTTP should be done using XML - i.e. that's probably how it would
> be done if the protocol were reinvented today ;-)

What, even more bloat?

java-programmer >> HTTP creep In comp.lang.java.advocacy, Tim Tyler
<email***@***.com>
wrote
on Wed, 7 Jul 2004 10:16:11 GMT
<email***@***.com>:
> HTTP should be done using XML - i.e. that's probably how it would
> be done if the protocol were reinvented today ;-)

Erm...XML might be used in the *payload*, although some proposals
are suggesting that major document sections can be grouped using
XML tags in the ESB using a publish-subscribe system (either JMS
or XSLT-based).

--
#191, email***@***.com
It's still legal to go .sigless.

java-programmer >> HTTP creep The Ghost In The Machine wrote:

> > Yuck! You don't apply the same principles to your source code, so why
> > should HTTP headers be any different? Ironically, it is probably the
> > non-abbreviated human-readable format, that enabled you to prematurely
> > argue for obfuscating it in the name of optimization.
>
> Maybe, but if 10M browsers are out there each generating 1K
> per request, that's an extra 100Gb of bandwidth. If one
> browses an average of a page a minute, that's 1.6 Gb/s
> just throwing headers around.

And if 192 people download the 52,315KB Java 1.4.2_05 SDK, they've used more
bandwidth then that. Dunno about you, but I downloaded two copies of Java 1.4.2_04
(one for work, one for home), I downloaded two copies of Java 1.4.2_03, I
downloaded a single copy of Java 1.4.2_05 from Sun, but FTPed it to work, so I
used the same overall network bandwidth.

So there's ... 6 of those 192 transfers of over 50MB of data (not to mention my
downloads of 1.4.2_01 and 02, and every SDK prior to that).

> Of course, compared to the viruses that's probably pretty paltry. :-)

It's paltry compared to downloads from CVS refreshes of FreeBSD packages and "apt"
and "yum" auto-download installations of RPMs and what "up2date" does and Windows
Update downloads and people downloading Fedora and Firefox and Mozilla and Camino
and people harvesting newsgroups and Kazaa and Bittorrents and streaming audio and
video.

Considering I can make almost 1,000,000 Web page requests with 1KB headers before
I "waste" as much bandwidth as I used when I downloaded FreeBSD 5.2.1 (just disc1
and disc2 ISOs mind you, not the miniinst or bootonly images) I think our
attention is better spent elsewhere.

So sure, let's form a standards group to design a new specification for HTTP that
supports all of Roedy's suggestions, I'm all for it. Except I'd suggest those
people have better things to do with their time then redesign a protocol that
"wastes" 0.01% (or less) of the bandwidth on the Internet.

java-programmer >> HTTP creep On Wed, 07 Jul 2004 16:16:17 GMT, Grant Wagner
<email***@***.com> wrote or quoted :

>And if 192 people download the 52,315KB Java 1.4.2_05 SDK, they've used more
>bandwidth then that. Dunno about you, but I downloaded two copies of Java 1.4.2_04
>(one for work, one for home), I downloaded two copies of Java 1.4.2_03, I
>downloaded a single copy of Java 1.4.2_05 from Sun, but FTPed it to work, so I
>used the same overall network bandwidth.

I think that bandwidth too is a problem, not an excuse for further
waste. There should be a delta scheme to download just the
differences between versions automatically depending on what you have
now.

We have got into a Roman orgy style of thinking that waste is GOOD.
Waste for the sake of waste. It is the thinking behind our species'
ecological irresponsibility.
--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.