Tricking Websites with cookies, etc.

[wfsaxton]

Hey all,

I'm currently writing an application that reads and parses web pages from a site. I am a member of this site, however, the server doesn't know this when my application talks to it. I've checked my cookies for all of the cookies set by the site, and used:

URL myURL = new URL("http://www.blahblah.com");
URLConnection myConn;

myConn = myURL.openConnection();
myConn.setRequestProperty("Cookie", "A=foo; B=bar");
myConn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461)");
myConn.setRequestProperty("Referer", "http://www.blahblah.com");
myConn.connect();



When I print out the inputstream, however, it get the page that says I'm not a member.

I'm guess I'm missing more stuff to set on my URLConnection. Anyone have any ideas what the server could be looking for so it stops shutting me out?

[ractoc]

Have you tried calling the login page from your script and writing the response from a successful login?
You might be able to get more info on what the server expects that way.

[wfsaxton]

What do you mean by that?

[ractoc]

well, when the server uses client side cookies, data is sent to the client (you) when you request to login.
this is the same info you need to send to the server when you want to fake being logged in

[wfsaxton]

Okay, I tried out your recommendation and I got stuck again

I downloaded httpunit because it makes this coding more intuitive. Here is my current code (including actual site links):

import com.meterware.httpunit.*;
import com.meterware.httpunit.WebResponse;
import java.io.*;

public class Main {
   public static void main(String[] args) throws Exception {
  
       String url = "http://comments.myspace.com/index.cfm?fuseaction=user.HomeComments&friendID=358314";
       
       HttpUnitOptions.setExceptionsThrownOnScriptError(false);

       WebConversation wc = new WebConversation();
       ClientProperties cp = wc.getClientProperties();       
       cp.setUserAgent("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.6) Gecko/20050223 Firefox/1.0.1");      
       cp.setAcceptCookies(true);
       
       wc.putCookie("IID","DD02BA2F%2DD58F%2D4225%2D9C79%2D8F58B4C0EECC");       
       wc.putCookie("MYUSERINFO","");
       wc.putCookie("USTID","107842582");
       
       System.out.println("---Attempting to access comment page---");
       WebResponse r = wc.getResponse(url);       
       
       System.out.println(r);

       WebLink loginlink = r.getLinkWith("Click Here");
       loginlink.click();
       System.out.println("---Going to login page---");
       WebResponse login = wc.getCurrentPage();
       System.out.println(login);
       
       WebForm forms[] = login.getForms();

       forms[1].setParameter("email","foo@bar.com");
       forms[1].setParameter("password","password");
       SubmitButton buttons[] = forms[1].getSubmitButtons();
       
       System.out.println("---Attempting to login---");           
       WebResponse loggingin = forms[1].submit();
       
   }
}

And here is the output. I end up getting a "Connection Refused" Error which I don't understand. I printed out the headers @ each step:

---Attempting to access comment page---
HttpWebResponse [url=http://signup.myspace.com/index.cfm?fuseaction=join.step1mod&nextPage=fuseaction%3Duser%2EHomeComments%26friendID%3D 358314; headers=
CONTENT-TYPE: text/html
CONNECTION: close
X-POWERED-BY: ASP.NET
SERVER: Microsoft-IIS/5.0
PAGE-COMPLETION-STATUS: Normal
PAGE-COMPLETION-STATUS: Normal
PAGE-COMPLETION-STATUS: Normal
DATE: Mon, 04 Apr 2005 03:04:31 GMT
SET-COOKIE: IID=DD02BA2F%2DD58F%2D4225%2D9C79%2D8F58B4C0EECC; expires=Sat, 02-Jul-2005 20:04:32 GMT; path=/; domain=.myspace.com;
SET-COOKIE: MYUSERINFO=; expires=Fri, 04-Mar-2005 19:04:31 GMT; path=/; domain=.myspace.com;
SET-COOKIE: USTID=109134959; expires=Mon, 04-Apr-2005 20:04:32 GMT; path=/; domain=.myspace.com;
P3P: CP="CAO DSP COR CUR CON OUR STP UNI" policyref=http://www.flowgo.com/w3c/p3p.xml
P3P: CP="CAO DSP COR CUR CON OUR STP UNI" policyref=http://derdb.mspaceads.com/w3c/p3p.xml ]
---Going to login page---
HttpWebResponse [url=http://signup.myspace.com/index.cfm?fuseaction=login&Mytoken=20050403200431; headers=
CONTENT-TYPE: text/html
CONNECTION: close
X-POWERED-BY: ASP.NET
SERVER: Microsoft-IIS/5.0
PAGE-COMPLETION-STATUS: Normal
PAGE-COMPLETION-STATUS: Normal
PAGE-COMPLETION-STATUS: Normal
DATE: Mon, 04 Apr 2005 03:05:32 GMT
SET-COOKIE: MYUSERINFO=; expires=Fri, 04-Mar-2005 19:05:32 GMT; path=/; domain=.myspace.com;
P3P: CP="CAO DSP COR CUR CON OUR STP UNI" policyref=http://www.flowgo.com/w3c/p3p.xml
P3P: CP="CAO DSP COR CUR CON OUR STP UNI" policyref=http://derdb.mspaceads.com/w3c/p3p.xml ]
---Attempting to login---
java.io.IOException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:602)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:272)
at com.meterware.httpunit.HttpWebResponse.readResponseHeader(HttpWebResponse.java:162)
at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:200)
at com.meterware.httpunit.HttpWebResponse.<init>(HttpWebResponse.java:56)
at com.meterware.httpunit.HttpWebResponse.<init>(HttpWebResponse.java:67)
at com.meterware.httpunit.WebConversation.newResponse(WebConversation.java:76)
at com.meterware.httpunit.WebWindow.getResource(WebWindow.java:164)
at com.meterware.httpunit.ParsedHTML.getIncludedScript(ParsedHTML.java:357)
at com.meterware.httpunit.HTMLPage$1.getIncludedScript(HTMLPage.java:257)
at com.meterware.httpunit.parsing.NekoDOMParser.getIncludedScript(NekoDOMParser.java:105)
at com.meterware.httpunit.parsing.ScriptFilter.startElement(ScriptFilter.java:92)
at org.cyberneko.html.filters.DefaultFilter.startElement(Unknown Source)
at org.cyberneko.html.filters.NamespaceBinder.startElement(Unknown Source)
at org.cyberneko.html.HTMLTagBalancer.callStartElement(Unknown Source)
at org.cyberneko.html.HTMLTagBalancer.startElement(Unknown Source)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanStartElement(Unknown Source)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(Unknown Source)
at org.cyberneko.html.HTMLScanner.scanDocument(Unknown Source)
at org.cyberneko.html.HTMLConfiguration.parse(Unknown Source)
at org.cyberneko.html.HTMLConfiguration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at com.meterware.httpunit.parsing.NekoHTMLParser.parse(NekoHTMLParser.java:41)
at com.meterware.httpunit.HTMLPage.parse(HTMLPage.java:255)
at com.meterware.httpunit.WebResponse.getReceivedPage(WebResponse.java:1126)
at com.meterware.httpunit.WebResponse$Scriptable.load(WebResponse.java:688)
at com.meterware.httpunit.javascript.JavaScript.load(JavaScript.java:89)
at com.meterware.httpunit.javascript.JavaScriptEngineFactory.load(JavaScriptEngineFactory.jav a:58)
at com.meterware.httpunit.RequestContext.runScripts(RequestContext.java:44)
at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:122)
at com.meterware.httpunit.WebWindow.sendRequest(WebWindow.java:110)
at com.meterware.httpunit.WebRequestSource.submitRequest(WebRequestSource.java:253)
at com.meterware.httpunit.WebRequestSource.submitRequest(WebRequestSource.java:232)
at com.meterware.httpunit.WebForm.submitRequest(WebForm.java:96)
at com.meterware.httpunit.WebForm.doFormSubmit(WebForm.java:107)
at com.meterware.httpunit.WebForm.submit(WebForm.java:69)
at com.meterware.httpunit.WebForm.submit(WebForm.java:58)
at newmain.Main.main(Main.java:52)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
at java.net.Socket.connect(Socket.java:452)
at java.net.Socket.connect(Socket.java:402)
at sun.net.NetworkClient.doConnect(NetworkClient.java:139)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:402)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:618)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:306)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:267)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:320)
at sun.net.www.http.HttpClient.New(HttpClient.java:315)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:521)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:498)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:626)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:1504)
at com.meterware.httpunit.HttpWebResponse.loadHeaders(HttpWebResponse.java:216)
at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:198)
... 35 more
java.lang.RuntimeException: java.io.IOException
at com.meterware.httpunit.WebResponse.getReceivedPage(WebResponse.java:1130)
at com.meterware.httpunit.WebResponse$Scriptable.load(WebResponse.java:688)
at com.meterware.httpunit.javascript.JavaScript.load(JavaScript.java:89)
at com.meterware.httpunit.javascript.JavaScriptEngineFactory.load(JavaScriptEngineFactory.jav a:58)
at com.meterware.httpunit.RequestContext.runScripts(RequestContext.java:44)
at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:122)
at com.meterware.httpunit.WebWindow.sendRequest(WebWindow.java:110)
at com.meterware.httpunit.WebRequestSource.submitRequest(WebRequestSource.java:253)
at com.meterware.httpunit.WebRequestSource.submitRequest(WebRequestSource.java:232)
at com.meterware.httpunit.WebForm.submitRequest(WebForm.java:96)
at com.meterware.httpunit.WebForm.doFormSubmit(WebForm.java:107)
at com.meterware.httpunit.WebForm.submit(WebForm.java:69)
at com.meterware.httpunit.WebForm.submit(WebForm.java:58)
at newmain.Main.main(Main.java:52)
Exception in thread "main"
Java Result: 1