Diadem is a good recommendation. However, databases were built to handle very large amounts of data. I've seen databases that have such huge contents that it boggles the mind. Taking in test data where each record has about 20 fields, and taking in about 1000 records per day over the course of years. The thing is HUGE! I once asked a database expert how can you sift through all that data in such short time. His explanation was that each record is indexed for easy and quick find. A binary tree or somthing like that is used as the search algorithm. Computers execute at speeds of a million instructions per second and greater. It doesn't take very long for a database engine to search through millions of records. So don't be afraid to store everything in a database, no matter how huge it may seem to you. Remember, lots of big corporations hold large amounts of data in databases. Can you imaging the size of databases used by the US government, the IRS, banks, the CIA, insurance companies, etc.? Your database will be pale in size compared to those. You just need the hard drive space to hold it all.
