[nzlug] Data processing.....

[Nevyn]

On 5/28/07, Cliff Pratt <enkidu at cliffp.com> wrote:
> Andras Farago wrote:
> > Hi guys, What software can you recommend to process a database/txt
> > file/whatever with approx 20 million rows, 25 fields in each row? I
> > need to perform only very simple queries like "SELECT a, Sum(b) FROM
> > data GROUP BY a ORDER BY Sum(b) DESC;". Another important option is
> > the time, I can't wait all day to get the result.
> >
> If you loaded it into a database you could build an index on it and get
> quick results that way. Though loading it and building the index could
> take some time.
>
> Cheers,
>
> Cliff

I'd have to say that the best approach here would be chose a platform
whether it be postgre or mysql and learn the little tips and tricks
for making things fast.

Of course a database is going to take up more room - there's more data
to store. Loading things generally take longer too initially just
because there's more data to generate. However, once all that data has
been generated and stored, you've now got speedy queries due to that
data that took oh so long to generate.

You can turn off indexing while loading in the data but the indexing
has to be done at some point.