Session Tracking and HTTP Data Extraction using java.net
Hi, I have a question for those of you who have experience with java.net, I would really appreciate your help.
I'm trying to extract data from web pages using Java, I understand that if a web page is static then we can use HttpURLConnection to get inputstream and therefore read its contents.
However the page I need to extract is dynamicly generated by a HTTP request query (using either GET or POST method), let's say the query is a HTTP GET request: http://www.myhost.com/search.jsp?par...is¶m2=that , if I paste this query into a brower it would work just fine and return the results I want, but when I call this query through a URLConnection object it fails to retrieve the data because it complains the session is invalid, this doesn't seem to work even when I set the cookie into my request... :-(
So has anyone done anything similar to this before? Can you give me any leads on this? Or do you know how to monitor the HTTP traffic goes in and out of your local machine so I can somehow send the request and intercept the response?
Many thanks for your help!
Just got the problem sorted. The reason it didn't work before is because I did not handle the page redirection, a http request send to the server gets redirected 6 times and everytime I need to pass back all the cookies obtained from previous redirection responses. Found an excellent article on cookie management and web query, http://www.javaworld.com/javaworld/j...26-cookie.html , this guy really has done it all.
Top DevX Stories
Easy Web Services with SQL Server 2005 HTTP Endpoints
JavaOne 2005: Java Platform Roadmap Focuses on Ease of Development, Sun Focuses on the "Free" in F.O.S.S.
Wed Yourself to UML with the Power of Associations
Microsoft to Add AJAX Capabilities to ASP.NET
IBM's Cloudscape Versus MySQL