Running server-side Java on FreeBSD in production environments  
Author Message
brian





PostPosted: 2003-12-3 5:14:00 Top

java-programmer, Running server-side Java on FreeBSD in production environments
So, I've seen a few mentions here of folks using Java server apps on
FreeBSD in production. I've been very happy to watch the progress by the
Java team on the native JDKs from the sidelines, as well as the
improvements to libc_r and such that have been MFC'd, but I'm curious as
to whether the consensus on this list is that it's ready to be used for
production. Anyone want to share good success or horror stories? We're
trying to determine whether the apache.org box can start to run our own
software. :) Speed is definitely less important than proper functioning.

Brian
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
freebsd





PostPosted: 2003-12-3 5:47:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments I've actually been running my site with FreeBSD Java since July of 2002,
first with the green-threads JDK 1.3 with OpenJIT, then with the native
threads 1.3 with Hotspot, and now with 1.4.2-p5.

My observation is that for server-side work (which is the only thing I've
tested) I run into more problems because of MY code than because of the
JVM. I haven't experienced any kind of crash of the JVM since 1.4.1-p4.
The fork fixes for p5 have improved stability quite a bit in my
environment, or at least I get the impression they have. I used to
experience Resin occasionally going apeshit; that hasn't happened since I
went to 1.4.2-p5.

I cannot stress enough the need of making sure your code is sound before
you run it in production. You CAN bring down a servlet container with the
code you run in it if you're not careful. The easiest way to do that is
to keep too many things in session or application scope or otherwise
manage to allocate large objects with long lifespans that use up all your
available heap. (voice of experience)

I also suggest running the JVM for a servlet container with these options:

-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

The -server flag tends to make server startup and class reloading take a
bit longer as I think it makes Hotspot a bit more aggressive about what it
compiles ahead of time. The other flags I got from a Sun paper on garbage
collection: http://java.sun.com/docs/hotspot/gc1.4.2/

Using incremental/low-pause garbage collection is pretty important for a
servlet container, imo; I'd generally rather add a few ms to each request
than have requests take 10-15 seconds occasionally when the JVM needs to
do GC.

One sorta hokey/kludgy thing I do as well is to perform a nightly restart
of all my JVMs. I don't know if I still need to do that with 1.4.2-p5.
The main reason I've been doing it at all is that memory seems to get
fragmented or possibly some objects are just never getting garbage
collected, so the JVM processes were using more CPU and memory as time
went by. I haven't verified whether this is still problematic in p5.

I definitely recommend arranging servlet containers into a primary/backup
JVM situation if your container allows it. I also suggest load balancing
across two containers for performance (it seems like 2 containers handle
load better under high-load condition than one container with twice as
many threads, which makes sense if you think about how threads and
processes differ).

My arrangement has 3 containers; two load-balanced primary JVMs and one
backup JVM.

When restarting your containers and when starting them the first time, I
suggest staggering the loading because of the amount of CPU the -server
switch causes the JVM to use as it starts. I start and restart mine with
a 60-second sleep interval between them. Otherwise, you end up thrashing
during startup, which can actually cause startup to fail because, in the
case of Resin anyway, the minder process can't contact the servlet
container because it's still loading because the CPU is pegged trying to
start up many containers at once.

Another thing I recommend is writing a "ping" process for your containers.
I have one I wrote for Resin that checks each minute to see that Resin's
"srun" service is working and is capable of loading a page (including
configurable timeouts, retries, etc). If there's no answer, it
automatically restarts the container and sends me an email. Though it's
quite rare that this actually does any work, it does happen periodically
and it's nice to have things restarted for me in the middle of the night
;)

That's all I can think of at the moment. Let me know if you'd like more
detail about anything in particular. I'm currently running morons.org,
ln-s.net, and go2rider.org with the FreeBSD JVM.

Nick

On Tue, 2 Dec 2003, Brian Behlendorf wrote:

>
> So, I've seen a few mentions here of folks using Java server apps on
> FreeBSD in production. I've been very happy to watch the progress by the
> Java team on the native JDKs from the sidelines, as well as the
> improvements to libc_r and such that have been MFC'd, but I'm curious as
> to whether the consensus on this list is that it's ready to be used for
> production. Anyone want to share good success or horror stories? We're
> trying to determine whether the apache.org box can start to run our own
> software. :) Speed is definitely less important than proper functioning.
>
> Brian
> _______________________________________________
> email***@***.com mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-java
> To unsubscribe, send any mail to "email***@***.com"
>

--
"The aptly-named morons.org is an obscenity-laced screed..."
-- Robert P. Lockwood, Catholic League director of research
Nick Johnson, version 2.1 http://web.morons.org/
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
jgrosch





PostPosted: 2003-12-3 6:03:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments
--fUYQa+Pmc3FrFX/N
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Dec 02, 2003 at 01:12:37PM -0800, Brian Behlendorf wrote:
>=20
> So, I've seen a few mentions here of folks using Java server apps on
> FreeBSD in production. I've been very happy to watch the progress by the
> Java team on the native JDKs from the sidelines, as well as the
> improvements to libc_r and such that have been MFC'd, but I'm curious as
> to whether the consensus on this list is that it's ready to be used for
> production. Anyone want to share good success or horror stories? We're
> trying to determine whether the apache.org box can start to run our own
> software. :) Speed is definitely less important than proper functioning.
>=20
> Brian

=46rom my limited experience I can say that FreeBSD 5.1 with Java 1.4.1 and
Tomcat 4.1.27 seems to work pretty well. Not real fast though. When threads
are done it should be really good. I've also used ant, eclipse, jboss,
Log4j and the other tools from the Jakarta project and they mostly
work. The only time I have had a problem it was from a mistake on my
part. Oh, jBoss and Eclipse are not from the Jakarta project but they
interface well.


Josef

--=20
Josef Grosch | Another day closer to a | FreeBSD 5.1
email***@***.com | Micro$oft free world | Berkeley, Ca.

--fUYQa+Pmc3FrFX/N
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----

iD8DBQE/zQmHy8prLS1GYSERAsceAKCYdckURK8hXkflkGXYifEBtKY8iACgyFoa
r/e1X7giQX1Ea26B4Wh2abc=
=Hm3o
-----END PGP SIGNATURE-----

--fUYQa+Pmc3FrFX/N--
 
 
brent





PostPosted: 2003-12-3 7:06:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments [2003-12-02 13:12] Brian Behlendorf said:
|
| So, I've seen a few mentions here of folks using Java server apps on
| FreeBSD in production. I've been very happy to watch the progress by the
| Java team on the native JDKs from the sidelines, as well as the
| improvements to libc_r and such that have been MFC'd, but I'm curious as
| to whether the consensus on this list is that it's ready to be used for
| production. Anyone want to share good success or horror stories? We're
| trying to determine whether the apache.org box can start to run our own
| software. :) Speed is definitely less important than proper functioning.

Here's my observation of running tomcat and jboss on -STABLE for
over a year now.

1.3.1-p8 (w/ hotspot)
stability : excellent
performance : sluggish (seemed to be about %40 slower than 1.4.1
when I last compared the two on some
ab(8) tests against tomcat)

1.4.2-p5 (w/ hotspot)
stability : very good/excellent (_may_ have seen this version
cause me to have to kill(1)
the jave process when trying
to shutdown tomcat, but cannot
be certain it wasn't something
in my code)
performance : very good (no formal comparison, but seems to
match performance per-MHz as sun's
jdk on debian/linux-2.4)

I would certainly deploy a native java on FreeBSD (4.9-STABLE)
for nearly-critical tomcat applications. That said, there are some
features which are not complete and/or busted. If the application
runs at all, I'd feel pretty confident it will keep on running.

cheers.
Brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
past





PostPosted: 2003-12-3 18:33:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments On Tuesday 02 December 2003 23:12, Brian Behlendorf wrote:
> So, I've seen a few mentions here of folks using Java server apps on
> FreeBSD in production. I've been very happy to watch the progress by the
> Java team on the native JDKs from the sidelines, as well as the
> improvements to libc_r and such that have been MFC'd, but I'm curious as
> to whether the consensus on this list is that it's ready to be used for
> production. Anyone want to share good success or horror stories? We're
> trying to determine whether the apache.org box can start to run our own
> software. :) Speed is definitely less important than proper functioning.
>
> Brian

I have deployed three servlet-based apps using tomcat & velocity in a
farm-like environment and have never experienced any problems. The
applications require jdk 1.4, so I have followed that path all along, mainly
in -stable machines (and -current for compiling/testing before deployment).

Cheers,
--
Panagiotis Astithas
Electrical & Computer Engineer, PhD
Network Management Center
National Technical University of Athens, Greece

_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
jbq





PostPosted: 2003-12-3 19:10:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments
--f61P+fpdnY2FZS1u
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

* Nick Johnson:

> One sorta hokey/kludgy thing I do as well is to perform a nightly
> restart of all my JVMs. I don't know if I still need to do that with
> 1.4.2-p5. The main reason I've been doing it at all is that memory
> seems to get fragmented or possibly some objects are just never
> getting garbage collected, so the JVM processes were using more CPU
> and memory as time went by. I haven't verified whether this is still
> problematic in p5.

You can prevent the OutOfMemory errors by preventing the class reloading
=66rom happening. On resin: class-update-interval=3D'31536000'.

Cheers,
--=20
Jean-Baptiste Quenot
http://caraldi.com/jbq/

--f61P+fpdnY2FZS1u
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/zcRS9xx3BCMc9gsRArtXAKCjBURI2qp+KDjFKJZ0m8tHpQZaxwCcCdLZ
m+aCaaz29er7la/wJ2efaXU=
=Th8L
-----END PGP SIGNATURE-----

--f61P+fpdnY2FZS1u--
 
 
sheldonh





PostPosted: 2003-12-3 22:43:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments On (2003/12/02 13:12), Brian Behlendorf wrote:

> Anyone want to share good success or horror stories? We're
> trying to determine whether the apache.org box can start to run our own
> software. :) Speed is definitely less important than proper functioning.

For me, the issue isn't so much with speed as it is for support of
heavily-threaded applications:

http://starjuice.net/2003_09_01_starjuice_archive.html#106267661433883916

Ciao,
Sheldon.
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
glewis





PostPosted: 2003-12-4 4:45:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments Hi Brian,

On Tue, Dec 02, 2003 at 01:12:37PM -0800, Brian Behlendorf wrote:
> So, I've seen a few mentions here of folks using Java server apps on
> FreeBSD in production. I've been very happy to watch the progress by the
> Java team on the native JDKs from the sidelines, as well as the
> improvements to libc_r and such that have been MFC'd, but I'm curious as
> to whether the consensus on this list is that it's ready to be used for
> production. Anyone want to share good success or horror stories? We're
> trying to determine whether the apache.org box can start to run our own
> software. :) Speed is definitely less important than proper functioning.

FWIW, in terms of proper functioning, 1.3.1 is compliant with Sun's
test suite and 1.4.2 almost is (unofficially). The one case where 1.4.2
isn't compliant is currently under discussion with Sun and is unlikely
to affect you unless you're using multicast in a mixed IPv4 and IPv6
environment.

Of the two, 1.4.2 is significantly faster thanks to HotSpot.

If you decide to go ahead, I'm willing to support you as much as I
can since I think apache.org running some Java servlet containers on
FreeBSD would be a great advertisement for Java on BSD :).

--
Greg Lewis Email : email***@***.com
Eyes Beyond Web : http://www.eyesbeyond.com
Information Technology FreeBSD : email***@***.com

_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
brian





PostPosted: 2003-12-4 4:51:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments
Thanks for everyone's comments. It's pretty encouraging, though not for
something anticipated to be extremely high volume. Since the apps we're
looking at are probably not going to have 2000 concurrent threads,
sounds like it's worth at least trying.

Brian

_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
peter.schuller





PostPosted: 2003-12-4 7:11:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments > FWIW, in terms of proper functioning, 1.3.1 is compliant with Sun's
> test suite and 1.4.2 almost is (unofficially). The one case where 1.4.2
> isn't compliant is currently under discussion with Sun and is unlikely
> to affect you unless you're using multicast in a mixed IPv4 and IPv6
> environment.

Is there somewhere one can read the unofficial details? A malinglist perhaps
that I don't know about? :)

--
/ Peter Schuller, InfiDyne Technologies HB

PGP userID: 0xE9758B7D or 'Peter Schuller <email***@***.com>'
Key retrieval: Send an E-Mail to email***@***.com
E-Mail: email***@***.com Web: http://www.scode.org


_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
ari





PostPosted: 2003-12-6 17:44:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments I'm coming in a little late on this, but I can report great success
with the FreeBSD JDK 1.3.1 running with WebObjects on 4.7 FreeBSD. I am
right this minutes in the middle of configuring a second deployment
server with 1.4.1 and am trying to decide between the IBM JDK and the
native FreeBSD one.

Since the IBM JDK gets such good reviews for speed I thought I'd try
it. I'll try and report on it when I get some time to test.

Anyhow, 1.3.1 FreeBSD JDK has been running in production for over a
year under WebObjects. It averages 150,000 page views per month all
served from Java based pages. Not a single hiccup.

Ari Maniatis



On 03/12/2003, at 8:12 AM, Brian Behlendorf wrote:

> So, I've seen a few mentions here of folks using Java server apps on
> FreeBSD in production.



-------------------------->
ish group pty ltd
7 Darghan St Glebe 2037 Australia
phone +61 2 9660 1400 fax +61 2 9660 7400
http www.ish.com.au | email email***@***.com
PGP fingerprint 08 57 20 4B 80 69 59 E2 A9 BF 2D 48 C2 20 0C C8

_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
brent





PostPosted: 2003-12-7 21:22:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments [2003-12-03 16:41] Sheldon Hearn said:
| On (2003/12/02 13:12), Brian Behlendorf wrote:
|
| > Anyone want to share good success or horror stories? We're
| > trying to determine whether the apache.org box can start to run our own
| > software. :) Speed is definitely less important than proper functioning.
|
| For me, the issue isn't so much with speed as it is for support of
| heavily-threaded applications:
|
| http://starjuice.net/2003_09_01_starjuice_archive.html#106267661433883916

Out of curiousity, I just ran the volano benchmark against the native
jdk1.4.2. More info on the benchmark as well as results from other
systems is available at http://www.volano.com/benchmarks.html.

server: 10.0.0.1
client: 10.0.0.1 (unfortunately, my linux box couldn't build up
more than about 600 client connections before
complaining of the stack being too small.)
OS: 4.9-STABLE (vintage 2 Dec 2003)
jvm args: -server -Xmx512M -Xss96k
hardware: 500Mhz pIII 768MB (10.0.0.1 is on a 3COM 3C905C-TX card)
sysctl settings:
jail.sysvipc_allowed=1
jail.socket_unixiproute_only=1
jail.set_hostname_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.somaxconn=1024
kern.ipc.nmbclusters=32768
vfs.vmiodirenable=1
net.inet.tcp.msl=10000 # cuts TIME_WAIT down to about 5s
kern.maxfiles=16384


results - not too bad :-)

1000 concurrent connections/threads.
test-1.log: [Sun Dec 7 07:36:51 EST 2003] Test started.
test-1.log: Average throughput = 1255 messages per second

2000 concurrent connections/threads.
test-2.log: [Sun Dec 7 07:46:11 EST 2003] Test started.
test-2.log: Average throughput = 960 messages per second

[At about 2500 connections, the server side died from "Out of memory"]

snippet from vmstat while benchmark running:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad4 in sy cs us sy id
2 0 0 2268924 213740 3 0 0 0 613 0 0 0 307 3657 76 33 67 0
1 0 0 2268924 213676 17 0 0 0 520 0 4 0 378 4452 46 37 63 0
2 0 0 2268924 213184 130 0 0 0 752 0 0 0 311 1072 106 59 41 0


I suspect having the client on another system would yield _much_ better
results. If I figure out how to get my linux box to handle the client
task, I'll report the difference in performance.

cheers.
Brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
brent





PostPosted: 2003-12-7 22:27:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments [2003-12-07 08:20] Brent Verner said:
| [2003-12-03 16:41] Sheldon Hearn said:
| | On (2003/12/02 13:12), Brian Behlendorf wrote:
| |
| | > Anyone want to share good success or horror stories? We're
| | > trying to determine whether the apache.org box can start to run our own
| | > software. :) Speed is definitely less important than proper functioning.
| |
| | For me, the issue isn't so much with speed as it is for support of
| | heavily-threaded applications:
| |
| | http://starjuice.net/2003_09_01_starjuice_archive.html#106267661433883916
|
| Out of curiousity, I just ran the volano benchmark against the native
| jdk1.4.2. More info on the benchmark as well as results from other
| systems is available at http://www.volano.com/benchmarks.html.

...

| I suspect having the client on another system would yield _much_ better
| results. If I figure out how to get my linux box to handle the client
| task, I'll report the difference in performance.

No luck with the linux client, so here are some runs at 500
concurrent connections for comparison's sake. Results from linux
box listed first. Looks like running on the same box cuts the
results just about in half, as expected.


sleepy:~/volano
brent$ /usr/local/jdk1.4.2/bin/java -Xmx512M -Xss256k -Dinstall.root=. -Dcatalina.home=. -cp .:lib/bootstrap.jar:lib/catalina.jar:lib/naming-common.jar:lib/naming-resources.jar:lib/servlet.jar:lib/servlets-common.jar:lib/servlets-default.jar:lib/volanochat-server.jar:lib/xerces.jar COM.volano.Mark -count 10 -host 10.0.0.1 -rooms 25
...
Creating room number 25 ...
500 connections so far.
Java heap: 2170 KB in use, 7588 KB available (29% in use).
Running the test ...
Test complete.

VolanoMark version = 2.5.0.9
Messages sent = 5000
Messages received = 95000
Total messages = 100000
Elapsed time = 31.628 seconds
Average throughput = 3162 messages per second



mutt:~/volano
brent$ /usr/local/jdk1.4.2/bin/java -server -Xmx64M -Dinstall.root=. -Dcatalina.home=. -cp .:lib/bootstrap.jar:lib/catalina.jar:lib/naming-common.jar:lib/naming-resources.jar:lib/servlet.jar:lib/servlets-common.jar:lib/servlets-default.jar:lib/volanochat-server.jar:lib/xerces.jar COM.volano.Mark -count 10 -host 10.0.0.1 -rooms 25
...
Creating room number 25 ...
500 connections so far.
Java heap: 2148 KB in use, 7612 KB available (28% in use).
Running the test ...
Test complete.

VolanoMark version = 2.5.0.9
Messages sent = 5000
Messages received = 95000
Total messages = 100000
Elapsed time = 66.585 seconds
Average throughput = 1502 messages per second


cheers.
Brent

--
"Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing." -- Duane Allman
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"
 
 
sheldonh





PostPosted: 2003-12-8 20:39:00 Top

java-programmer >> Running server-side Java on FreeBSD in production environments On (2003/12/06 20:42), Aristedes Maniatis wrote:

> Anyhow, 1.3.1 FreeBSD JDK has been running in production for over a
> year under WebObjects. It averages 150,000 page views per month all
> served from Java based pages. Not a single hiccup.

Stable, sure. But not necessarily fast. Even if you assume only 20
business days in the month, that's still only an average of 5 views per
minute. :-)

Ciao,
Sheldon.
_______________________________________________
email***@***.com mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "email***@***.com"