AVR-Chat

> I do not plan to have one CPU to search the whole
10Tb > I was planing on using 100 parallel CPU's each
looking > after 100Gb of ram.

Speaking from experience (In a past life, I configured
and tuned IBM SP/2 machines...) a data set that is
this large could only be processed using a massively
parallel super computer. Something along the line of
an IBM SP/2 (If speed is your need and price is no
option, you'll want 128 bit RISC cpu's...) or a
beowulf cluster composed of off the shelf hardware (If
speed is good, but economy is better...). 

Specifically, the type of data sets you have described
cannot even be measured (Which is the starting point
for processing any data set this large...), much less
processed by any currently available cpu that is
operating as a distinctly autonomous computational
device. Well, I take that back, it could be done, but
even the accuracy of the measurements of a data set
that large would be suspect and the time it took to
realize an answer would most likely render the data
usesless (The measurement would be suspect because the
time it took to measure it would prohibit redundancy).
Further, even if you were only calculating the MD5
checksum of a data set this large on a single cpu, the
process would run for a very_long_time: Months
(Plural).

Regards

Marvin Dickens
Alpharetta, Georgia  USA


=====
Registered Linux User No. 80253
If you use linux, get counted at: 
http://www.linuxcounter.org


		
__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - Send 10MB messages!
http://promotions.yahoo.com/new_mail