Download a webpage that contains javascript  
Author Message
googleaddress





PostPosted: 2004-9-14 4:20:00 Top

java-programmer, Download a webpage that contains javascript I am trying to download the html page at

http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1

Using the code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

But all I am getting is


<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>


I'm guessing that when the web brower meets this code it executes the
javascript


location.replace("http://www.dreamteamfc.com")


And somehow loads the web page.


How do I go about downloading the actual web page that gets displayed
in a normal web browser.




Any help appeciated!


pat
 
Chander





PostPosted: 2004-9-14 6:15:00 Top

java-programmer >> Download a webpage that contains javascript Have you checked its behavious whenyou try opening on Browser..? it
redirects you to someother page and checks if you are logged in.. etc etc..

The way you are doing is not the proper way you have to follow the RFC
for HTTP1.1 for the same... the procedure used by you might work for
some sites but not for all the sites....

Cheers

http://www.ziksa.net
email: email***@***.com

Patrick wrote:
> I am trying to download the html page at
>
> http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1
>
> Using the code
>
> public class DownloadWebPage
> { public static void main (String[] args) throws IOException
> {
> URL url = new
> URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
> BufferedReader webRead = new BufferedReader(new
> InputStreamReader(url.openStream()));
> String line;
> while ((line = webRead.readLine()) != null)
> {
> System.out.println(line);
> }
>
> }
> }
>
> But all I am getting is
>
>
> <HTML><HEAD><SCRIPT
> LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
>
>
> I'm guessing that when the web brower meets this code it executes the
> javascript
>
>
> location.replace("http://www.dreamteamfc.com")
>
>
> And somehow loads the web page.
>
>
> How do I go about downloading the actual web page that gets displayed
> in a normal web browser.
>
>
>
>
> Any help appeciated!
>
>
> pat
 
googleaddress





PostPosted: 2004-9-14 22:50:00 Top

java-programmer >> Download a webpage that contains javascript I had a mistake in my previous post, it should have read:

If I navigate to
http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH


And click on a link which calls the follow javascript function
"javascript:dt_pop('PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "


A new page pops up, the browser say its url is

http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167


Now with my code

public class DownloadWebPage
{ public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
BufferedReader webRead = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
while ((line = webRead.readLine()) != null)
{
System.out.println(line);
}

}
}

I can download the page at

http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH


But I cannot download the page at

http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167


That the javascript function

"javascript:dt_pop('PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167',
'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "


In the page

http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH

calls.



When I try to download

http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167

With my code, all i get is

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>




So how do i go about downloading the contents of the html page that is
in the window that is displayed when the javascript function above is
called?


regards,

pat

Chander <email***@***.com> wrote in message news:<7_o1d.6844$email***@***.com>...
> Have you checked its behavious whenyou try opening on Browser..? it
> redirects you to someother page and checks if you are logged in.. etc etc..
>
> The way you are doing is not the proper way you have to follow the RFC
> for HTTP1.1 for the same... the procedure used by you might work for
> some sites but not for all the sites....
>
> Cheers
>
> http://www.ziksa.net
> email: email***@***.com
>
> Patrick wrote:
> > I am trying to download the html page at
> >
> > http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1
> >
> > Using the code
> >
> > public class DownloadWebPage
> > { public static void main (String[] args) throws IOException
> > {
> > URL url = new
> > URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
> > BufferedReader webRead = new BufferedReader(new
> > InputStreamReader(url.openStream()));
> > String line;
> > while ((line = webRead.readLine()) != null)
> > {
> > System.out.println(line);
> > }
> >
> > }
> > }
> >
> > But all I am getting is
> >
> >
> > <HTML><HEAD><SCRIPT
> > LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
> >
> >
> > I'm guessing that when the web brower meets this code it executes the
> > javascript
> >
> >
> > location.replace("http://www.dreamteamfc.com")
> >
> >
> > And somehow loads the web page.
> >
> >
> > How do I go about downloading the actual web page that gets displayed
> > in a normal web browser.
> >
> >
> >
> >
> > Any help appeciated!
> >
> >
> > pat
 
 
Chander





PostPosted: 2004-9-15 11:33:00 Top

java-programmer >> Download a webpage that contains javascript As I menatined earlier read the rfc for http 1.1 when you are requesting
a page firsdt you get is the header you get information for size of the
data to follow, if server has responded as error, what type of error..
or the page is to be redirected, etc.. as per these information you are
to read the following lines or resend the request to the other page as
direction in the header..

Cheers
Patrick wrote:
> I had a mistake in my previous post, it should have read:
>
> If I navigate to
> http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
>
>
> And click on a link which calls the follow javascript function
> "javascript:dt_pop('PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167',
> 'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
>
>
> A new page pops up, the browser say its url is
>
> http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
>
>
> Now with my code
>
> public class DownloadWebPage
> { public static void main (String[] args) throws IOException
> {
> URL url = new
> URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
> BufferedReader webRead = new BufferedReader(new
> InputStreamReader(url.openStream()));
> String line;
> while ((line = webRead.readLine()) != null)
> {
> System.out.println(line);
> }
>
> }
> }
>
> I can download the page at
>
> http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
>
>
> But I cannot download the page at
>
> http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
>
>
> That the javascript function
>
> "javascript:dt_pop('PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167',
> 'remote', 610, 550, 10, 10, 'no', 'yes', 'no', 'no'); "
>
>
> In the page
>
> http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
>
> calls.
>
>
>
> When I try to download
>
> http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
>
> With my code, all i get is
>
> <HTML><HEAD><SCRIPT
> LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
>
>
>
>
> So how do i go about downloading the contents of the html page that is
> in the window that is displayed when the javascript function above is
> called?
>
>
> regards,
>
> pat
>
> Chander <email***@***.com> wrote in message news:<7_o1d.6844$email***@***.com>...
>
>>Have you checked its behavious whenyou try opening on Browser..? it
>>redirects you to someother page and checks if you are logged in.. etc etc..
>>
>>The way you are doing is not the proper way you have to follow the RFC
>>for HTTP1.1 for the same... the procedure used by you might work for
>>some sites but not for all the sites....
>>
>>Cheers
>>
>>http://www.ziksa.net
>>email: email***@***.com
>>
>>Patrick wrote:
>>
>>>I am trying to download the html page at
>>>
>>> http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1
>>>
>>>Using the code
>>>
>>> public class DownloadWebPage
>>> { public static void main (String[] args) throws IOException
>>> {
>>> URL url = new
>>>URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1");
>>> BufferedReader webRead = new BufferedReader(new
>>>InputStreamReader(url.openStream()));
>>> String line;
>>> while ((line = webRead.readLine()) != null)
>>> {
>>> System.out.println(line);
>>> }
>>>
>>> }
>>> }
>>>
>>>But all I am getting is
>>>
>>>
>>> <HTML><HEAD><SCRIPT
>>>LANGUAGE="JAVASCRIPT">location.replace("http://www.dreamteamfc.com");</SCRIPT></HEAD></HTML>
>>>
>>>
>>>I'm guessing that when the web brower meets this code it executes the
>>>javascript
>>>
>>>
>>> location.replace("http://www.dreamteamfc.com")
>>>
>>>
>>>And somehow loads the web page.
>>>
>>>
>>>How do I go about downloading the actual web page that gets displayed
>>>in a normal web browser.
>>>
>>>
>>>
>>>
>>>Any help appeciated!
>>>
>>>
>>>pat
 
 
googleaddress





PostPosted: 2004-9-15 22:38:00 Top

java-programmer >> Download a webpage that contains javascript Ok, I am starting to understand what is going on now a little better.


I noticed that when I access

http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH

The server sends a cookie. And then when I access

http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167

I get the table of players and their respective points.

I used the following code

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167");
URLConnection uc = url.openConnection();
System.out.println(uc.getHeaderField("Set-Cookie"));
}


To get the cookie, which was

CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
22:25:46 GMT; Path=/

I think
CF_HA, is just a unique identifier, a variable which in incremented
by the server for each new client
Domain, is just the domain
expires, is just the expiry date
Path, hmm dunno

Now I hardcoded the cookie into the code, with a valid expiry date

public static void main (String[] args) throws IOException
{
URL url = new
URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167");
URLConnection uc = url.openConnection();

String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
uc.setRequestProperty("cookie",cookie);
int i = 0;

while ((i = uc.getInputStream().read()) != -1)
{ System.out.print((char) i);
}
}


Now, when I run this code I get the following error


Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1133)
at Test.main(Test.java:46)


Is the following method

uc.setRequestProperty("cookie",cookie);

The correct way to send the cookie to the server?


Any pointers on what I should try next, I'm not quite sure where to
look.


pat

Chander <email***@***.com> wrote in message news:<aKO1d.12626$email***@***.com>...
> As I menatined earlier read the rfc for http 1.1 when you are requesting
> a page firsdt you get is the header you get information for size of the
> data to follow, if server has responded as error, what type of error..
> or the page is to be redirected, etc.. as per these information you are
> to read the following lines or resend the request to the other page as
> direction in the header..
>
> Cheers
 
 
Mark Marcus





PostPosted: 2004-9-21 10:36:00 Top

java-programmer >> Download a webpage that contains javascript On 15 Sep 2004 07:38:22 -0700, email***@***.com (Patrick)
wrote:

>Ok, I am starting to understand what is going on now a little better.
>
>
>I noticed that when I access
>
> http://www.dreamteamfc.com/dtfc04/servlet/OpenFSELogin?homename=dtfc04&language=ENGLISH
>
>The server sends a cookie. And then when I access
>
> http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
>
>I get the table of players and their respective points.
>
>I used the following code
>
> public static void main (String[] args) throws IOException
> {
> URL url = new
>URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167");
> URLConnection uc = url.openConnection();
> System.out.println(uc.getHeaderField("Set-Cookie"));
> }
>
>
>To get the cookie, which was
>
> CF_HA=2415676698; Domain=.dreamteamfc.com; expires=Tue, 14-Sep-04
>22:25:46 GMT; Path=/
>
>I think
> CF_HA, is just a unique identifier, a variable which in incremented
>by the server for each new client
> Domain, is just the domain
> expires, is just the expiry date
> Path, hmm dunno
>
>Now I hardcoded the cookie into the code, with a valid expiry date
>
> public static void main (String[] args) throws IOException
> {
> URL url = new
>URL("http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167");
> URLConnection uc = url.openConnection();
>
> String cookie = "CF_HA=2415676698; Domain=.dreamteamfc.com;
>expires=Tue, 14-Sep-04 22:25:46 GMT; Path=/";
> uc.setRequestProperty("cookie",cookie);
> int i = 0;
>
> while ((i = uc.getInputStream().read()) != -1)
> { System.out.print((char) i);
> }
> }
>
>
>Now, when I run this code I get the following error
>
>
>Exception in thread "main" java.io.IOException: Server returned HTTP
>response code: 400 for URL:
>http://www.dreamteamfc.com/dtfc04/servlet/PostPlayerList?catidx=1&title=GOALKEEPERS&gameid=167
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1133)
> at Test.main(Test.java:46)
>
>
>Is the following method
>
> uc.setRequestProperty("cookie",cookie);
>
>The correct way to send the cookie to the server?
>
>
>Any pointers on what I should try next, I'm not quite sure where to
>look.
>
>
>pat
>
>Chander <email***@***.com> wrote in message news:<aKO1d.12626$email***@***.com>...
>> As I menatined earlier read the rfc for http 1.1 when you are requesting
>> a page firsdt you get is the header you get information for size of the
>> data to follow, if server has responded as error, what type of error..
>> or the page is to be redirected, etc.. as per these information you are
>> to read the following lines or resend the request to the other page as
>> direction in the header..
>>
>> Cheers


Well, I agree with Chander that you need to read the RFC on HTTP 1.1
(RFC 2068) which can be found at
http://www.ietf.org/rfc/rfc2068.txt?number=2068. I'd give you more
detailed advice, but I think a lot of it would be speculation (my
programmer friend is sitting next to me, so these are his words)

So here is the speculation. The issue that might be the cause of your
woes. In order for you to set properties in a URLConnection, the
properties MUST be set before you call getInputStream() or getHeader()