Improvements that I can do to my database comparing?


DevX Home    Today's Headlines   Articles Archive   Tip Bank   Forums   

Results 1 to 6 of 6

Thread: Improvements that I can do to my database comparing?

  1. #1
    Blake Guest

    Improvements that I can do to my database comparing?


    Hi,
    I need some help, I am comparing 2 very large databases ranging from 2.6
    million entries to 23 million entries. What I have my comparer program do
    is it compares the data from the old database to the new database and any
    entries that arent found in the new database it reports them to a text file.
    My problem is when it finds an entry thats not in the new database the program
    "pauses" because its searching till the end of the 2.5 million entries(or
    22 million) So when an entry isnt found in the new database I experience
    a "pause" of around 5 seconds for the 2.5 million entries and 2 minutes for
    the 23 million entries. I use Find Next to search and compare the 2 databases
    and when an entry in the old database is found in the new database that entry
    is removed from the new database, this keeps my searching speed at a constant
    rate. So as more entries are removed the "pause" time shortens because the
    number of entries to search for in the new database decrease each time an
    entry in the old database is found in the new database. Does anyone have
    any suggestions of what I can do to shorten my "pause"? I understand this
    may be a little confusing to understand, but any suggestions would be helpful.
    Thank you!

  2. #2
    Paul Clapham Guest

    Re: Improvements that I can do to my database comparing?

    It would help to know how you are doing this. Are you doing an SQL join on
    the two tables? Or do you have a merge-sort algorithm in your program that
    tries to read the two tables in parallel? Are the tables you are reading in
    the same sequence, and do you have indexes on that sequence?

    Blake <Flitcher@aol.com> wrote in message news:3a9d7454@news.devx.com...
    >
    > Hi,
    > I need some help, I am comparing 2 very large databases ranging from 2.6
    > million entries to 23 million entries. What I have my comparer program do
    > is it compares the data from the old database to the new database and any
    > entries that arent found in the new database it reports them to a text

    file.
    > My problem is when it finds an entry thats not in the new database the

    program
    > "pauses" because its searching till the end of the 2.5 million entries(or
    > 22 million) So when an entry isnt found in the new database I experience
    > a "pause" of around 5 seconds for the 2.5 million entries and 2 minutes

    for
    > the 23 million entries. I use Find Next to search and compare the 2

    databases
    > and when an entry in the old database is found in the new database that

    entry
    > is removed from the new database, this keeps my searching speed at a

    constant
    > rate. So as more entries are removed the "pause" time shortens because the
    > number of entries to search for in the new database decrease each time an
    > entry in the old database is found in the new database. Does anyone have
    > any suggestions of what I can do to shorten my "pause"? I understand this
    > may be a little confusing to understand, but any suggestions would be

    helpful.
    > Thank you!




  3. #3
    Blake Guest

    Re: Improvements that I can do to my database comparing?


    Hi,
    The databases are in JetEngine. And the entries in the 2 databases are not
    always in the same rows, and everyday new entries are added to the new database.
    Hope this helps. Thanks!
    "Paul Clapham" <pclapham@core-mark.com> wrote:
    >It would help to know how you are doing this. Are you doing an SQL join

    on
    >the two tables? Or do you have a merge-sort algorithm in your program that
    >tries to read the two tables in parallel? Are the tables you are reading

    in
    >the same sequence, and do you have indexes on that sequence?
    >
    >Blake <Flitcher@aol.com> wrote in message news:3a9d7454@news.devx.com...
    >>
    >> Hi,
    >> I need some help, I am comparing 2 very large databases ranging from 2.6
    >> million entries to 23 million entries. What I have my comparer program

    do
    >> is it compares the data from the old database to the new database and

    any
    >> entries that arent found in the new database it reports them to a text

    >file.
    >> My problem is when it finds an entry thats not in the new database the

    >program
    >> "pauses" because its searching till the end of the 2.5 million entries(or
    >> 22 million) So when an entry isnt found in the new database I experience
    >> a "pause" of around 5 seconds for the 2.5 million entries and 2 minutes

    >for
    >> the 23 million entries. I use Find Next to search and compare the 2

    >databases
    >> and when an entry in the old database is found in the new database that

    >entry
    >> is removed from the new database, this keeps my searching speed at a

    >constant
    >> rate. So as more entries are removed the "pause" time shortens because

    the
    >> number of entries to search for in the new database decrease each time

    an
    >> entry in the old database is found in the new database. Does anyone have
    >> any suggestions of what I can do to shorten my "pause"? I understand this
    >> may be a little confusing to understand, but any suggestions would be

    >helpful.
    >> Thank you!

    >
    >



  4. #4
    Paul Clapham Guest

    Re: Improvements that I can do to my database comparing?

    So you don't have indexes on these tables?

    Blake <Flitcher@aol.com> wrote in message news:3a9d91f1$1@news.devx.com...
    >
    > Hi,
    > The databases are in JetEngine. And the entries in the 2 databases are not
    > always in the same rows, and everyday new entries are added to the new

    database.
    > Hope this helps. Thanks!
    > "Paul Clapham" <pclapham@core-mark.com> wrote:
    > >It would help to know how you are doing this. Are you doing an SQL join

    > on
    > >the two tables? Or do you have a merge-sort algorithm in your program

    that
    > >tries to read the two tables in parallel? Are the tables you are reading

    > in
    > >the same sequence, and do you have indexes on that sequence?





  5. #5
    blake Guest

    Re: Improvements that I can do to my database comparing?


    Hi,
    No I dont have indexes on these tables.
    "Paul Clapham" <pclapham@core-mark.com> wrote:
    >So you don't have indexes on these tables?
    >
    >Blake <Flitcher@aol.com> wrote in message news:3a9d91f1$1@news.devx.com...
    >>
    >> Hi,
    >> The databases are in JetEngine. And the entries in the 2 databases are

    not
    >> always in the same rows, and everyday new entries are added to the new

    >database.
    >> Hope this helps. Thanks!
    >> "Paul Clapham" <pclapham@core-mark.com> wrote:
    >> >It would help to know how you are doing this. Are you doing an SQL join

    >> on
    >> >the two tables? Or do you have a merge-sort algorithm in your program

    >that
    >> >tries to read the two tables in parallel? Are the tables you are reading

    >> in
    >> >the same sequence, and do you have indexes on that sequence?

    >
    >
    >



  6. #6
    Paul Clapham Guest

    Re: Improvements that I can do to my database comparing?

    And are you reading the tables in sequence by the keys you're using to
    compare the two tables, or are you reading them in random sequence?
    blake <flitcher@aol.com> wrote in message news:3a9ec654$1@news.devx.com...
    >
    > Hi,
    > No I dont have indexes on these tables.
    > "Paul Clapham" <pclapham@core-mark.com> wrote:
    > >So you don't have indexes on these tables?
    > >
    > >Blake <Flitcher@aol.com> wrote in message

    news:3a9d91f1$1@news.devx.com...
    > >>
    > >> Hi,
    > >> The databases are in JetEngine. And the entries in the 2 databases are

    > not
    > >> always in the same rows, and everyday new entries are added to the new

    > >database.
    > >> Hope this helps. Thanks!
    > >> "Paul Clapham" <pclapham@core-mark.com> wrote:
    > >> >It would help to know how you are doing this. Are you doing an SQL

    join
    > >> on
    > >> >the two tables? Or do you have a merge-sort algorithm in your program

    > >that
    > >> >tries to read the two tables in parallel? Are the tables you are

    reading
    > >> in
    > >> >the same sequence, and do you have indexes on that sequence?

    > >
    > >
    > >

    >




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center
 
 
FAQ
Latest Articles
Java
.NET
XML
Database
Enterprise
Questions? Contact us.
C++
Web Development
Wireless
Latest Tips
Open Source


   Development Centers

   -- Android Development Center
   -- Cloud Development Project Center
   -- HTML5 Development Center
   -- Windows Mobile Development Center