Improvements that I can do to my database comparing?

[Blake]

Hi,
I need some help, I am comparing 2 very large databases ranging from 2.6
million entries to 23 million entries. What I have my comparer program do
is it compares the data from the old database to the new database and any
entries that arent found in the new database it reports them to a text file.
My problem is when it finds an entry thats not in the new database the program
"pauses" because its searching till the end of the 2.5 million entries(or
22 million) So when an entry isnt found in the new database I experience
a "pause" of around 5 seconds for the 2.5 million entries and 2 minutes for
the 23 million entries. I use Find Next to search and compare the 2 databases
and when an entry in the old database is found in the new database that entry
is removed from the new database, this keeps my searching speed at a constant
rate. So as more entries are removed the "pause" time shortens because the
number of entries to search for in the new database decrease each time an
entry in the old database is found in the new database. Does anyone have
any suggestions of what I can do to shorten my "pause"? I understand this
may be a little confusing to understand, but any suggestions would be helpful.
Thank you!

[Paul Clapham]

It would help to know how you are doing this. Are you doing an SQL join on
the two tables? Or do you have a merge-sort algorithm in your program that
tries to read the two tables in parallel? Are the tables you are reading in
the same sequence, and do you have indexes on that sequence?

Blake <Flitcher@aol.com> wrote in message news:3a9d7454@news.devx.com...
>
> Hi,
> I need some help, I am comparing 2 very large databases ranging from 2.6
> million entries to 23 million entries. What I have my comparer program do
> is it compares the data from the old database to the new database and any
> entries that arent found in the new database it reports them to a text
file.
> My problem is when it finds an entry thats not in the new database the
program
> "pauses" because its searching till the end of the 2.5 million entries(or
> 22 million) So when an entry isnt found in the new database I experience
> a "pause" of around 5 seconds for the 2.5 million entries and 2 minutes
for
> the 23 million entries. I use Find Next to search and compare the 2
databases
> and when an entry in the old database is found in the new database that
entry
> is removed from the new database, this keeps my searching speed at a
constant
> rate. So as more entries are removed the "pause" time shortens because the
> number of entries to search for in the new database decrease each time an
> entry in the old database is found in the new database. Does anyone have
> any suggestions of what I can do to shorten my "pause"? I understand this
> may be a little confusing to understand, but any suggestions would be
helpful.
> Thank you!

[Blake]

Hi,
The databases are in JetEngine. And the entries in the 2 databases are not
always in the same rows, and everyday new entries are added to the new database.
Hope this helps. Thanks!
"Paul Clapham" <pclapham@core-mark.com> wrote:
>It would help to know how you are doing this. Are you doing an SQL join
on
>the two tables? Or do you have a merge-sort algorithm in your program that
>tries to read the two tables in parallel? Are the tables you are reading
in
>the same sequence, and do you have indexes on that sequence?
>
>Blake <Flitcher@aol.com> wrote in message news:3a9d7454@news.devx.com...
>>
>> Hi,
>> I need some help, I am comparing 2 very large databases ranging from 2.6
>> million entries to 23 million entries. What I have my comparer program
do
>> is it compares the data from the old database to the new database and
any
>> entries that arent found in the new database it reports them to a text
>file.
>> My problem is when it finds an entry thats not in the new database the
>program
>> "pauses" because its searching till the end of the 2.5 million entries(or
>> 22 million) So when an entry isnt found in the new database I experience
>> a "pause" of around 5 seconds for the 2.5 million entries and 2 minutes
>for
>> the 23 million entries. I use Find Next to search and compare the 2
>databases
>> and when an entry in the old database is found in the new database that
>entry
>> is removed from the new database, this keeps my searching speed at a
>constant
>> rate. So as more entries are removed the "pause" time shortens because
the
>> number of entries to search for in the new database decrease each time
an
>> entry in the old database is found in the new database. Does anyone have
>> any suggestions of what I can do to shorten my "pause"? I understand this
>> may be a little confusing to understand, but any suggestions would be
>helpful.
>> Thank you!
>
>

[Paul Clapham]

So you don't have indexes on these tables?

Blake <Flitcher@aol.com> wrote in message news:3a9d91f1$1@news.devx.com...
>
> Hi,
> The databases are in JetEngine. And the entries in the 2 databases are not
> always in the same rows, and everyday new entries are added to the new
database.
> Hope this helps. Thanks!
> "Paul Clapham" <pclapham@core-mark.com> wrote:
> >It would help to know how you are doing this. Are you doing an SQL join
> on
> >the two tables? Or do you have a merge-sort algorithm in your program
that
> >tries to read the two tables in parallel? Are the tables you are reading
> in
> >the same sequence, and do you have indexes on that sequence?

[blake]

Hi,
No I dont have indexes on these tables.
"Paul Clapham" <pclapham@core-mark.com> wrote:
>So you don't have indexes on these tables?
>
>Blake <Flitcher@aol.com> wrote in message news:3a9d91f1$1@news.devx.com...
>>
>> Hi,
>> The databases are in JetEngine. And the entries in the 2 databases are
not
>> always in the same rows, and everyday new entries are added to the new
>database.
>> Hope this helps. Thanks!
>> "Paul Clapham" <pclapham@core-mark.com> wrote:
>> >It would help to know how you are doing this. Are you doing an SQL join
>> on
>> >the two tables? Or do you have a merge-sort algorithm in your program
>that
>> >tries to read the two tables in parallel? Are the tables you are reading
>> in
>> >the same sequence, and do you have indexes on that sequence?
>
>
>

[Paul Clapham]

And are you reading the tables in sequence by the keys you're using to
compare the two tables, or are you reading them in random sequence?
blake <flitcher@aol.com> wrote in message news:3a9ec654$1@news.devx.com...
>
> Hi,
> No I dont have indexes on these tables.
> "Paul Clapham" <pclapham@core-mark.com> wrote:
> >So you don't have indexes on these tables?
> >
> >Blake <Flitcher@aol.com> wrote in message
news:3a9d91f1$1@news.devx.com...
> >>
> >> Hi,
> >> The databases are in JetEngine. And the entries in the 2 databases are
> not
> >> always in the same rows, and everyday new entries are added to the new
> >database.
> >> Hope this helps. Thanks!
> >> "Paul Clapham" <pclapham@core-mark.com> wrote:
> >> >It would help to know how you are doing this. Are you doing an SQL
join
> >> on
> >> >the two tables? Or do you have a merge-sort algorithm in your program
> >that
> >> >tries to read the two tables in parallel? Are the tables you are
reading
> >> in
> >> >the same sequence, and do you have indexes on that sequence?
> >
> >
> >
>