Synchronized output stream sends data out of order, Possible?  
Author Message
dynaheir





PostPosted: 2007-1-18 19:50:00 Top

java-programmer, Synchronized output stream sends data out of order, Possible? Hi All,

I have a strange problem... I have > 1000 threads wanting to send data
through synchronized DataOutputStream wrapped around a sockets
OutputStream. Replies sent across the internet are received perfectly
in the sent order but across the router the stream is somehow
corrupted.

Is there something I need to set to stop this problem?

Please take a look a the code and corrupted output to see if I'm just
being a bit dumb... thanks

to send:
try {
synchronized(dos) {
System.out.println(getName() + " send response " + file + "
resp " + resp);
dos.writeUTF(file);
dos.writeByte(resp);
dos.flush();
}
} catch (Exception e) {
System.out.println(file + " response failed lm");
e.printStackTrace();
}

I checked to see that syncronisation was being obeyed, no other thread
accesses the stream until it's released by another so all bytes are
written to the stream in the desired order.

to receive:
try {
pause(); //synchronize on this and wait until is.dataAvailable()>0
String file = is.readUTF();
System.out.println("rec reply " + file + " rm");
while (!"done".equals(file)) { //done respone sent when all reply
threads terminate
pause();
int reply = is.readByte();
System.out.println("resp " + reply + "rm");
pause();
file = is.readUTF();
System.out.println("rec reply " + file + " rm");
}
System.out.println("done received rm");
} catch (Exception e) {
e.printStackTrace();
}

send output:
Thread-1207 send response 0002_01.dat resp 4
Thread-1208 send response 0002_02.dat resp 4
Thread-1209 send response 0003_01.dat resp 0
Thread-1210 send response 0003_02.dat resp 0
Thread-1211 send response 0005_01.dat resp 0
Thread-1212 send response 0005_02.dat resp 0
Thread-1213 send response 0006_01.dat resp 0
Thread-1214 send response 0006_02.dat resp 0
Thread-1215 send response 0007_01.dat resp 4
Thread-1216 send response 0008_01.dat resp 4
Thread-1217 send response 0008_02.dat resp 4
Thread-1218 send response 0008_03.dat resp 4
Thread-1219 send response 0009_01.dat resp 0
Thread-1220 send response 0009_02.dat resp 0
Thread-1221 send response 0010_01.dat resp 4
Thread-1222 send response 0010_02.dat resp 4
Thread-1223 send response 0011_01.dat resp 0
Thread-1224 send response 0011_02.dat resp 0
Thread-1225 send response 0012_01.dat resp 4
Thread-1226 send response 0012_01_1.dat resp 0
Thread-1227 send response 0012_02.dat resp 4
Thread-1228 send response 0012_02_1.dat resp 0
Thread-1229 send response 0013_01.dat resp 0
Thread-1230 send response 0013_02.dat resp 0
Thread-1231 send response 0014_01.dat resp 0
Thread-1232 send response 0014_02.dat resp 0
Thread-1233 send response 0015_01.dat resp 0
Thread-1234 send response 0015_02.dat resp 0
Thread-1235 send response 0016_01.dat resp 0
Thread-1236 send response 0016_02.dat resp 0
//end of output

Note the order of this
Thread-1216 send response 0008_01.dat resp 4
Thread-1217 send response 0008_02.dat resp 4
Thread-1218 send response 0008_03.dat resp 4
Thread-1219 send response 0009_01.dat resp 0
Thread-1220 send response 0009_02.dat resp 0
Thread-1221 send response 0010_01.dat resp 4
Thread-1222 send response 0010_02.dat resp 4
Thread-1223 send response 0011_01.dat resp 0
Thread-1224 send response 0011_02.dat resp 0
Thread-1225 send response 0012_01.dat resp 4
Thread-1226 send response 0012_01_1.dat resp 0
Thread-1227 send response 0012_02.dat resp 4
Thread-1228 send response 0012_02_1.dat resp 0

You'll see below that after 0008_01.dat and reponse as 4 is sent
0012_02_1.dat is next received but wasn't sent until much further down
the list. This corrupts the stream, which then appears to be in the
correct order there onwards. Following output is all read in as the
result of a single readUTF() operation

Receive output
rec reply 0002_01.dat rm
resp 4rm
rec reply 0002_02.dat rm
resp 4rm
rec reply 0003_01.dat rm
resp 0rm
rec reply 0003_02.dat rm
resp 0rm
rec reply 0005_01.dat rm
resp 0rm
rec reply 0005_02.dat rm
resp 0rm
rec reply 0006_01.dat rm
resp 0rm
rec reply 0006_02.dat rm
resp 0rm
rec reply 0007_01.dat rm
resp 4rm
rec reply 0008_01.dat rm
resp 4rm
rec reply 0012_02_1.dat rm
resp 0rm
rec reply 008_02.dat 0008_03.dat 0009_01.dat 0009_02.dat
0010_01.dat 0010_02.dat 0011_01.dat 0011_02.dat 0012_01.dat
0012_01_1.dat 0012_02.dat 0013_01.dat 0013_02.dat 0014_01.dat
0014_02.dat 0015_01.dat 0015_02.dat 0016_01.dat 0016_02.dat
0017_01.dat 0017_02.dat 0018_01.dat 0018_02.dat 0019_01.dat
0019_02.dat 0020_01.dat 0020_02.dat 0021_01.dat 0021_02.dat
0022_01.dat 0022_02.dat 0023_01.dat 0023_02.dat 0024_02.dat
0025_02.dat 0026_02.dat 0029_02.dat 0025_01.dat 0029_01.dat
0026_01.dat 0024_01.dat 0030_01.dat 0030_02.dat 0031_01.dat
0032_01.dat 0031_02.dat 0032_02.dat 0035_01.dat 0035_02.dat
0037_01.dat 0037_02.dat 0038_01.dat 0038_02.dat 0041_01.dat
0041_02.dat 0042_02.dat 0042_01.dat 0043_01.dat 0044_01.dat
0045_01.dat 0043_02.dat 0044_02.dat 0046_01.dat 0045_02.dat
0046_02.dat 0047_01.dat 0047_02.dat 0048_01.dat 0048_02.dat
0049_01.dat 0049_02.dat 0050_01.dat 0050_02.dat 0051_01.dat
0051_02.dat 0052_01.dat 0052_02.dat 0053_01.dat 0053_02.dat
0054_01.dat 0054_02.dat 0055_01.dat 0055_02.dat 0056_01.dat
0056_02.dat 0057_02_1.dat 0058_01.dat 0058_02.dat 0059_01.dat
0059_02.dat 0060_01.dat 0060_01_1.dat 0060_02.dat 0060_02_1.dat
0063_01.dat 0063_02.dat 0067_01.dat 0068_01.dat 0067_02.dat
0068_02.dat 0069_01.dat 0070_01.dat 0069_02.dat 0070_02.dat
0071_01.dat 0071_02.dat 0072_01.dat 0072_02.dat 0074_01.dat
0074_02.dat 0075_01.dat 0075_02.dat 0076_01.dat 0076_02.dat
0089_01.dat 0093_01.dat 0094_01.dat 0095_01.dat 0096_01.dat
0097_01.dat 0099_01.dat 0099_02.dat 0101_01.dat 0102_01.dat
0103_01.dat 0105_01.dat 0106_01.dat 0107_01.dat 0108_01.dat
0109_01.dat 0110_01.dat 0128_01.dat 0128_02.dat 0141_01.dat
0142_01.dat 0143_01.dat 0200_01.dat 0200_01_1.dat 0200_02.dat
0200_02_1.dat 0201_01.dat 0201_02.dat 0202_01.dat 0202_02.dat
0203_01.dat 0203_02.dat 0205_01.dat 0205_02.dat 0206_01.dat
0206_02.dat 0207_01.dat 0207_02.dat 0208_01.dat 0208_02.dat
0209_01.dat 0209_02.dat 0210_01.dat 0210_02.dat 0211_01.dat
0211_02.dat 0212_01.dat 0212_02.dat 0214_02.dat 0215_01.dat
0215_02.dat 0216_01.dat 0216_02.dat 0217_01.dat 0217_02.dat
0219_01.dat 0219_02.dat 0220_01.dat 0220_02.dat 0221_01.dat
0221_02.dat 0222_01.dat 0222_02.dat 0223_01.dat 0223_02.dat
0224_01.dat 0224_02.dat 0225_02.dat 0226_01.dat 0226_02.dat
0227_02.dat 0230_02.dat 0231_01.dat 0231_02.dat 0232_01.dat
0232_02.dat 0233_01.dat 0233_02.dat 0237_02.dat 0237_01.dat
0238_01.dat 0238_02.dat 0239_01.dat 0239_02.dat 0240_01.dat
0240_02.dat 0250_01.dat 0250_01_1.dat 0250_02.dat 0251_01. rm
resp 100rm

Any ideas would be helpful.

Thank you

 
Tom Hawtin





PostPosted: 2007-1-19 0:37:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? email***@***.com wrote:
>
> I have a strange problem... I have > 1000 threads wanting to send data
> through synchronized DataOutputStream wrapped around a sockets
> OutputStream. Replies sent across the internet are received perfectly
> in the sent order but across the router the stream is somehow
> corrupted.

Are you sure you are synchronising with the same instance of
DataOuputStream every time?

Does it happen consistently, or just after certain data values?

A highly unlikely problem is unsafe publication. That is to say passing
the DataOutputStream to another thread in a thread-unsafe manner. I have
never actually seen that cause a problem (and you probably need odd
hardware to demonstrate it).

Tom Hawtin
 
dynaheir





PostPosted: 2007-1-19 1:22:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? I wish it happened consistently... at least then I would have something
to go at.

I can confirm that the same instance of the DataOutputStream is passed
into all the reply threads so it should be synchronizing on the same
instance.

I'm not sure I understand your last point, "passing the
DataOutputStream to another thread in a thread-unsafe manner", how
would this be acheived? The thread doing the reply work is in fact an
inner class of an inner class... both of which extend thread.

public class Core extends thread {
when server socket accept returns, a new requestProcessor is created
with the socket

private inner class requestProcessor extends thread {
reads the input stream and creates a reply handler for each
request with the outputstream

private inner inner class reply {
processes recieved data and sends a reply
this is where the synchronized block occurs
}
}
}

thanks for your reply

 
 
Lew





PostPosted: 2007-1-19 7:07:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? email***@***.com wrote:
> public class Core extends thread {

Is "thread" your class or did you misspell the standard Java API class name
"Thread"?

> private inner class requestProcessor extends thread {

You should name classes with an initial upper-case letter. This is a matter of
convention, not syntax, but it strongly promotes understandability and
maintainability of your code. Sun has a Coding Conventions document that you
should read. (Feel free to put opening braces on their own line in the same
column as the closing brace, however.)

And there is no "inner" keyword.

> private inner inner class reply {

Ditto.

Are you clear on the distinction between "inner" and "nested" classes?

Why did you elect to make these classes inner, instead of top level?

- Lew
 
 
dynaheir





PostPosted: 2007-1-19 14:43:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? That's not important, the syntax is valid... netbeans has no problems
with the structure and compiles fine... the program operates perfectly
across the internet but not on the router to a close by machine.

During testing last night I found that if I force it to wait in the
synchronized block for 10 ms then this section also operates normally
on the LAN... I don't understand why it should be neccessary though.

 
 
Andrew Thompson





PostPosted: 2007-1-19 14:59:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? email***@***.com wrote:
> That's not important,

Not if your 'talking' to machines...

>...the syntax is valid...

..but we are not machines, so we would only
read code if we felt like it. The conventions that
Lew referred to, make it easier for a human to
read, and if you want other people to read it,
it is wise to stick to the convention.

Andrew T.

 
 
dynaheir





PostPosted: 2007-1-19 16:26:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? Fair enough, buit the correct operation of the program in one scenario
and not in another makes me think that there's not an issue with
program structure, more like something going wrong in the stream when
data is transfering rapidly on the lan compared to slowly across the
internet.

Are there any issues with transfering large amounts of data through the
socket streams in both directions at the same time? Meaning, should I
check that the input stream is not receiving before I start sending.
Could data being send in both directions be causing some sort of
unreported buffer overflow on something like that?

Thanks for your help

 
 
Lew





PostPosted: 2007-1-19 21:15:00 Top

java-programmer >> Synchronized output stream sends data out of order, Possible? email***@***.com wrote:
> private inner class requestProcessor

Lew wrote:
> there is no "inner" keyword.

email***@***.com wrote:
> That's not important, the syntax is valid... netbeans has no problems
> with the structure and compiles fine... the program operates perfectly
> across the internet but not on the router to a close by machine.

private inner class requestProcessor

The compiler chokes on "inner". It does not compile. It chokes twice on "inner
inner".

- Lew