Betty Crocker Genomics

It is one thing to eagerly follow new technologies and muse about their differences, it is quite different to be in the position of playing the game with real money. In the genome gold rush years it was decided we needed more computing power to deal with searching the torrent of DNA sequence data, and so we started looking at the three then-extant providers of specialized sequence analysis computers. But how to pick which one, with each costing as much as a house?

So, I designed a bake-off: a careful evaluation of the three machines. Since I was busy with other projects, I attempted to define strict protocols which each company would follow with their own instrument. The results would be delivered to me within a set timeframe along with pre-specified summary information. Based on this, I would decide which machines met the minimal standard and which was the best value.

Designing good rules for a bake-off is a difficult task. You really need to understand your problem, as you want the rules to ensure that you get the comparative data you need in all the areas you need it, with no ambiguity. You also want to avoid wasting time drafting or evaluating criteria that aren't important to your mission. Of key importance is to not unfairly prejudice the competition against any particular entry or technology -- every rule must address the business goal, the whole business goal, and nothing but the business goal.

Our bake-off was a success, and we did purchase a specialized computer which sounded like a jet engine when it ran (fans! it ran hot!) -- but it was tucked away where nobody would routinely hear it. The machine worked well until we no longer needed it, and then we retired it -- and not long after the manufacturer retired from the scene, presumably because most of their customers had followed the same path as us.

I'm thinking about this now because a prize competition has been announced for DNA sequencing, the Archon X Prize. This is the same organization which successfully spurred the private development of a sub-orbital space vehicle, SpaceShip One. For the Genome X Prize, the basic goal is to sequence 100 diploid human genomes in 10 days for $1 million.

A recent GenomeWeb article described some of the early thoughts about the rules for this grand, public bake-off. The challenges in simply defining the rules are immense, and one can reasonably ask how they will shape the technologies which are used.

First off, what exactly does it mean to sequence 100 human genomes in 1 week for $1 million? Do you have to actually assemble the data in that time frame, or is that just the time to generate raw reads & filter them for variations? Can I run the sequencer in a developing country, where the labor & real estate costs are low? Does the capital cost of the machine count in the calculation? What happens to the small gaps in the current genome? To mistakes in the current assembly? To structural polymorphisms? Are all errors weighted equally, and what level is tolerable? Does every single repeat need to be sequenced correctly?

The precise laying down of rules will significantly affect which technologies will have a good chance. Requiring that repeats be finished completely, for example, would tend to favor long read lengths. On the other hand, very high basepair accuracy standards might favor other technologies. Cost calculation methods can be subject to dispute (e.g. this letter from George Church's group).

One can also ask the question as to whether fully sequencing 100 genomes is the correct goal. For example, one might argue that sequencing all of the coding regions from a normal human cell will get most of the information at lower cost. Perhaps the goal should be to sequence the complete transcriptomes from 1000 individuals. Perhaps the metagenomics of human tumors is what we really should be shooting for -- with appropriate goals for extreme sensitivity.

Despite all these issues, one can only applaud the attempt. After all, Consumer Reports does not review genomics technologies! With luck, the Genome X Prize will spur a new round of investment in genomics technologies and new companies and applications. Which reminds me, if anyone has Virgin Galactic tickets they don't plan to use, I'd be happy to take them off your hands...

Long enough to cover the subject, short enough to be interesting.

That was the advice my 10th grade English teacher passed on when asked how much we should produce for a writing assignment. The context (a woman's skirt) he gave was risque enough to get a giggle from 10th graders of the 80's; probably the same joke would get him in serious hot water today -- unless perhaps he pointed out that the same applies for a man's kilt.

A letter in a recent Nature suggests that the same question that vexed me in my student days also bedevils the informatics world. The writer lodges a complaint against MIAME (Minimum Information About a Microarray Experiment), a standard for reporting the experimental context of a microarray experiment. MIAME attempts to capture some key information, such as what the samples are and what was done to them.

The letter writer's complaint is that this is all a fool's mission, as one cannot possibly capture all the key information, especially since what is key to record keeps changing. All reasonable points.

The solution proposed made me re-read the letter for a hint of satire, but I'm afraid they are dead serious.

How should we proceed? Reducing the costs of microarray technology so that experiments can be readily reproduced across laboratories seems a reasonable approach. Relying on minimal standards of annotation such as MIAME seems unreasonable, and should be abandoned.

.

At first, this just seems like good science. After all, the acid test in science is replication by an independent laboratory.

This utterly ignores two facts. First, by depositing annotated data in central databanks the data can be mined by researchers who don't have access to microarray gear. Second, most interesting microarray experiments involve either specialized manipulations (which only a few labs can do) or very precious limited samples (such as clinical ones); replication would be nice but just can't be done on those same samples.

This "the experiments will be too cheap to database" argument has come up before; I had it sent my way during a seminar in my graduate days. But, like electricity too cheap to meter, it is a tantalizing mirage which fades on close inspection.