Submitting to web pages


DevX Home    Today's Headlines   Articles Archive   Tip Bank   Forums   

Results 1 to 6 of 6

Thread: Submitting to web pages

  1. #1
    Andy Guest

    Submitting to web pages


    I think this is going to be either a can of worms or dead easy, but I can't
    find anything in my Java textbooks about this.......

    How can a Java application submit information into a webpage form and then
    read the results back as a file?

    For example - www.teletext.co.uk/finance has a text entry field on the right
    hand side which takes a stock code, such as DXNS. Conceptually, I want my
    program to imitate a user by entering a string into the entry field and pressing
    the "GO" link, and then read the resulting web page for the search's results.

    Entering a code in the above page's text field results in a very long URL
    for the next page the user sees - at the moment, I cut & paste those URLs
    into a file which my program reads and then accesses directly using "URL"
    and "InputStream" classes. This works fine, but it's not an elegant solution.

    Any advice much appreciated, or links to code which does this sort of thing.

    Andy (two months experience of Java programming, and loving it so far!)


  2. #2
    Paul Clapham Guest

    Re: Submitting to web pages

    I was going to tell you to create a URL and read from its input stream, but
    when I read more I saw you had already figured that out. So what's the
    inelegance you want to get rid of?

    PC2

    "Andy" <ajmassey@london.com> wrote in message
    news:3c05048d$1@147.208.176.211...
    >
    > I think this is going to be either a can of worms or dead easy, but I

    can't
    > find anything in my Java textbooks about this.......
    >
    > How can a Java application submit information into a webpage form and then
    > read the results back as a file?
    >
    > For example - www.teletext.co.uk/finance has a text entry field on the

    right
    > hand side which takes a stock code, such as DXNS. Conceptually, I want my
    > program to imitate a user by entering a string into the entry field and

    pressing
    > the "GO" link, and then read the resulting web page for the search's

    results.
    >
    > Entering a code in the above page's text field results in a very long URL
    > for the next page the user sees - at the moment, I cut & paste those URLs
    > into a file which my program reads and then accesses directly using "URL"
    > and "InputStream" classes. This works fine, but it's not an elegant

    solution.
    >
    > Any advice much appreciated, or links to code which does this sort of

    thing.
    >
    > Andy (two months experience of Java programming, and loving it so far!)
    >




  3. #3
    andy Guest

    Re: Submitting to web pages


    Hi Paul. Thanks for replying to my post -let me clarify the problem.

    Many web pages have text entry fields into which the user has to enter their
    search requirements etc and then press the "SUBMIT" button; the website then
    takes the user to a new URL based on the search engine's results.

    www.teletext.co.uk/finance is a simple example of this. Enter DXNS in the
    "Company name or symbol" field on the right of the screen and click "GO".
    You will be taken to another URL which displays a share price for the DXNS
    stock. (I'll refer to this page as "page 2")

    My problem is that I want a java program to imitate the action of entering
    data into these text entry fields and then to initiate the "GO" submission.

    Currently, I am manually cutting and pasting the URL from "page 2" above
    into a text file which my java program can read in and then go straight to
    that URL.

    Hence, this is BYPASSING the text-field entry stage rather than MIMICKING
    it. And this is not a very elegant method - there must be a way for a java
    program to mimic a user by "entering" data into search fields itself.

    I hope this clarifies my problem.

    Regards
    Andy





    "Paul Clapham" <pclapham@core-mark.com> wrote:
    >I was going to tell you to create a URL and read from its input stream,

    but
    >when I read more I saw you had already figured that out. So what's the
    >inelegance you want to get rid of?
    >
    >PC2
    >
    >"Andy" <ajmassey@london.com> wrote in message
    >news:3c05048d$1@147.208.176.211...
    >>
    >> I think this is going to be either a can of worms or dead easy, but I

    >can't
    >> find anything in my Java textbooks about this.......
    >>
    >> How can a Java application submit information into a webpage form and

    then
    >> read the results back as a file?
    >>
    >> For example - www.teletext.co.uk/finance has a text entry field on the

    >right
    >> hand side which takes a stock code, such as DXNS. Conceptually, I want

    my
    >> program to imitate a user by entering a string into the entry field and

    >pressing
    >> the "GO" link, and then read the resulting web page for the search's

    >results.
    >>
    >> Entering a code in the above page's text field results in a very long

    URL
    >> for the next page the user sees - at the moment, I cut & paste those URLs
    >> into a file which my program reads and then accesses directly using "URL"
    >> and "InputStream" classes. This works fine, but it's not an elegant

    >solution.
    >>
    >> Any advice much appreciated, or links to code which does this sort of

    >thing.
    >>
    >> Andy (two months experience of Java programming, and loving it so far!)
    >>

    >
    >



  4. #4
    Paul Clapham Guest

    Re: Submitting to web pages

    What's the point of the text file? Couldn't you just paste that URL
    directly into your Java program? It would look like this:

    URL dixons = new
    URL("http://www.teletext.co.uk/finance/sharedetail.asp?X=%8C%91%AA%A9%A5%CF%
    8D%CE%BE%87%BA%9F%BA%CA%87v%7F%80%85q%9E%B1%B8%8Cq%A1o%9Al%B1z%B2%A7%9B");

    PC2

    "andy" <andrew.massey@totalise.co.uk> wrote in message
    news:3c0c8646$1@147.208.176.211...
    >
    > Hi Paul. Thanks for replying to my post -let me clarify the problem.
    >
    > Many web pages have text entry fields into which the user has to enter

    their
    > search requirements etc and then press the "SUBMIT" button; the website

    then
    > takes the user to a new URL based on the search engine's results.
    >
    > www.teletext.co.uk/finance is a simple example of this. Enter DXNS in the
    > "Company name or symbol" field on the right of the screen and click "GO".
    > You will be taken to another URL which displays a share price for the DXNS
    > stock. (I'll refer to this page as "page 2")
    >
    > My problem is that I want a java program to imitate the action of entering
    > data into these text entry fields and then to initiate the "GO"

    submission.
    >
    > Currently, I am manually cutting and pasting the URL from "page 2" above
    > into a text file which my java program can read in and then go straight to
    > that URL.
    >
    > Hence, this is BYPASSING the text-field entry stage rather than MIMICKING
    > it. And this is not a very elegant method - there must be a way for a java
    > program to mimic a user by "entering" data into search fields itself.
    >
    > I hope this clarifies my problem.
    >
    > Regards
    > Andy
    >
    >
    >
    >
    >
    > "Paul Clapham" <pclapham@core-mark.com> wrote:
    > >I was going to tell you to create a URL and read from its input stream,

    > but
    > >when I read more I saw you had already figured that out. So what's the
    > >inelegance you want to get rid of?
    > >
    > >PC2
    > >
    > >"Andy" <ajmassey@london.com> wrote in message
    > >news:3c05048d$1@147.208.176.211...
    > >>
    > >> I think this is going to be either a can of worms or dead easy, but I

    > >can't
    > >> find anything in my Java textbooks about this.......
    > >>
    > >> How can a Java application submit information into a webpage form and

    > then
    > >> read the results back as a file?
    > >>
    > >> For example - www.teletext.co.uk/finance has a text entry field on the

    > >right
    > >> hand side which takes a stock code, such as DXNS. Conceptually, I want

    > my
    > >> program to imitate a user by entering a string into the entry field and

    > >pressing
    > >> the "GO" link, and then read the resulting web page for the search's

    > >results.
    > >>
    > >> Entering a code in the above page's text field results in a very long

    > URL
    > >> for the next page the user sees - at the moment, I cut & paste those

    URLs
    > >> into a file which my program reads and then accesses directly using

    "URL"
    > >> and "InputStream" classes. This works fine, but it's not an elegant

    > >solution.
    > >>
    > >> Any advice much appreciated, or links to code which does this sort of

    > >thing.
    > >>
    > >> Andy (two months experience of Java programming, and loving it so far!)
    > >>

    > >
    > >

    >




  5. #5
    Andy Guest

    Re: Submitting to web pages


    Indeed I could, but....

    a) in this specific example, for some reason the URLs stop working after
    a couple of days, so I'd have to keep editing and recompiling the source
    file.

    b) I'd like to find a general way for a program to mimic a user as described
    below, so I can write programs which can interrogate websites via the search
    text entry fields on their homepages etc.

    There MUST be a way of doing this.

    "Paul Clapham" <pclapham@core-mark.com> wrote:
    >What's the point of the text file? Couldn't you just paste that URL
    >directly into your Java program? It would look like this:
    >
    >URL dixons = new
    >URL("http://www.teletext.co.uk/finance/sharedetail.asp?X=%8C%91%AA%A9%A5%CF%
    >8D%CE%BE%87%BA%9F%BA%CA%87v%7F%80%85q%9E%B1%B8%8Cq%A1o%9Al%B1z%B2%A7%9B");
    >
    >PC2
    >
    >"andy" <andrew.massey@totalise.co.uk> wrote in message
    >news:3c0c8646$1@147.208.176.211...
    >>
    >> Hi Paul. Thanks for replying to my post -let me clarify the problem.
    >>
    >> Many web pages have text entry fields into which the user has to enter

    >their
    >> search requirements etc and then press the "SUBMIT" button; the website

    >then
    >> takes the user to a new URL based on the search engine's results.
    >>
    >> www.teletext.co.uk/finance is a simple example of this. Enter DXNS in

    the
    >> "Company name or symbol" field on the right of the screen and click "GO".
    >> You will be taken to another URL which displays a share price for the

    DXNS
    >> stock. (I'll refer to this page as "page 2")
    >>
    >> My problem is that I want a java program to imitate the action of entering
    >> data into these text entry fields and then to initiate the "GO"

    >submission.
    >>
    >> Currently, I am manually cutting and pasting the URL from "page 2" above
    >> into a text file which my java program can read in and then go straight

    to
    >> that URL.
    >>
    >> Hence, this is BYPASSING the text-field entry stage rather than MIMICKING
    >> it. And this is not a very elegant method - there must be a way for a

    java
    >> program to mimic a user by "entering" data into search fields itself.
    >>
    >> I hope this clarifies my problem.
    >>
    >> Regards
    >> Andy
    >>
    >>
    >>
    >>
    >>
    >> "Paul Clapham" <pclapham@core-mark.com> wrote:
    >> >I was going to tell you to create a URL and read from its input stream,

    >> but
    >> >when I read more I saw you had already figured that out. So what's the
    >> >inelegance you want to get rid of?
    >> >
    >> >PC2
    >> >
    >> >"Andy" <ajmassey@london.com> wrote in message
    >> >news:3c05048d$1@147.208.176.211...
    >> >>
    >> >> I think this is going to be either a can of worms or dead easy, but

    I
    >> >can't
    >> >> find anything in my Java textbooks about this.......
    >> >>
    >> >> How can a Java application submit information into a webpage form and

    >> then
    >> >> read the results back as a file?
    >> >>
    >> >> For example - www.teletext.co.uk/finance has a text entry field on

    the
    >> >right
    >> >> hand side which takes a stock code, such as DXNS. Conceptually, I want

    >> my
    >> >> program to imitate a user by entering a string into the entry field

    and
    >> >pressing
    >> >> the "GO" link, and then read the resulting web page for the search's
    >> >results.
    >> >>
    >> >> Entering a code in the above page's text field results in a very long

    >> URL
    >> >> for the next page the user sees - at the moment, I cut & paste those

    >URLs
    >> >> into a file which my program reads and then accesses directly using

    >"URL"
    >> >> and "InputStream" classes. This works fine, but it's not an elegant
    >> >solution.
    >> >>
    >> >> Any advice much appreciated, or links to code which does this sort

    of
    >> >thing.
    >> >>
    >> >> Andy (two months experience of Java programming, and loving it so far!)
    >> >>
    >> >
    >> >

    >>

    >
    >



  6. #6
    Paul Clapham Guest

    Re: Submitting to web pages


    "Andy" <andrew.massey@totalise.co.uk> wrote in message
    news:3c0e316e$1@147.208.176.211...
    >
    > Indeed I could, but....
    >
    > a) in this specific example, for some reason the URLs stop working after
    > a couple of days, so I'd have to keep editing and recompiling the source
    > file.


    You'll notice the URL doesn't say
    http://www.teletext.co.uk/finance/sh...il.asp?X=DXNS, which would be
    sufficient for the request, but something more complicated. If it changes
    periodically, that could be because the operators of the website don't want
    people to do what you are trying to do. After all, if you did that then you
    wouldn't see their advertising.

    > b) I'd like to find a general way for a program to mimic a user as

    described
    > below, so I can write programs which can interrogate websites via the

    search
    > text entry fields on their homepages etc.


    Presumably when you get a valid response from the server, you then have to
    rummage through the HTML and pick out the share price that the user's eyes
    would automatically see. When the website is redesigned, which probably
    happens a couple of times a year, the HTML changes and your program breaks,
    unless it is as clever as the user is. What you want is the latest industry
    hype, called "Web Services". The idea is, you send a message in a
    particular format to the server saying "Give me the stock price for DXNS"
    and it sends back a message saying "10.50". Stock prices are a common
    example given by people promoting Web Services, but I don't expect them to
    happen anytime soon (see what I said above about advertising).

    > There MUST be a way of doing this.


    Why? Not everything is possible.




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center
 
 
FAQ
Latest Articles
Java
.NET
XML
Database
Enterprise
Questions? Contact us.
C++
Web Development
Wireless
Latest Tips
Open Source


   Development Centers

   -- Android Development Center
   -- Cloud Development Project Center
   -- HTML5 Development Center
   -- Windows Mobile Development Center