Sunday, April 01, 2012

Announcing the Omics! Omics! Sequencer Evaluation Program

A serious issue which has been raised often both by myself and commenters is the challenge of properly evaluating sequencer performance prior to widespread commercial availability of a platform.  A number of roads have been frequently taken, but each have their issues.  Data generated by a company is always met with suspicion.  Alpha releases are often to academics connected to a company, which has the advantage that they are likely the most passionate about proving its worth, but the downside these users may not be viewed as fully objective.  Beta (or early release) sites are often less connected to a company, but by the time they publish their results through peer-reviewed journals it is long since commercial launch.  Conferences such as Marco Island help shorten the cycle time, but not entirely.  Early users are sometimes also suspected of having agreements which give the platform makers the power to squash bad data.   So how can a manufacturer gain "street cred" for their instrument
We at Omics! Omics! are excited to announce today a new service which complements these traditional routes to validating a new sequencing platform.  This offer is open to all sequencer platforms, whether they are the third (or fourth) chain terminator chemistry to hit the market or some exotic chemistry never before seen in the wild.  All are eligible for this exciting new offering.

The premise for the service is simple  All a manufacturer needs to do is deliver to Omics! Omics! (at an address we designate), one completely working instrument and all necessary accessories and reagents to run twenty samples.  We'll agree not to reverse engineer anything, but no other agreements are necessary (to avoid tainting us with the suspicion of collusion), and to ensure our independence absolutely no money changes hands.  We will then run samples of our choosing on the instrument, and report detailed statistics for every run in this space, eliminating the lag from traditional publication routes.  We'll also make publicly available the data from a representative subset of the runs.

What sort of samples will we run?  Well, it of course depends on the instrument's parameters (for example, is it intended for targeted or whole genome sequencing), but you can be sure we will find a variety of challenging samples.  Cheek swabs from various members of the team.  Heck, we'll even dig up the palatial grounds of Omics! Omics! central to find interesting samples.

But, as they say on TV, that's not all.  We have several variants on the program for platforms trying to demonstrate a little extra.  For example, if a manufacturer wishes to demonstrate their machine is suitable for field work, we can place it in our special test space which has no temperature control or even dust management.  At a push of a button, we can even expose the system to the outside air (don't worry; the access door is protected by rolling code technology).  Compact sequencers which will fit in the rear of our special test vehicle (and operate off standard 12V line) can be demonstrated to operate under difficult road conditions.  Offering a USB stick device?  We can put it through the wringer too!

As a special bonus, any manufacturers who register this month will be allowed to send us reagents sufficient for 50 samples!  Imagine the immense credibility from such a large, independent dataset.  Yet truly priceless, since we charge nothing for this service.  How could anyone resist such an offer?


Endre Major said...

I guess you can wait for a while till the companies to send you a few million dollars worth of equipment. I have already implemented a simulation web service that supports data generation for most platforms. It is not real data but should give an estimate on some basic platform differences and their effect on the results. See:

biohish said...

Hey Keith,
That is so avid of an enthusiasm, standing at the receiver's end of this, as someone who is going to be making a decision for which of the sequencers I will go for from among many selection options, how are we going to get access to the outcomes of such tests ? I mean, do we have to subscribe to view these findings or will it be publicly available ?

Keith Robison said...


Once a year it pays to pay attention to the posting date on my entry :-) But for all, including biohish, if anyone takes me up on the offer, I would publish it here with no strings attached.

Simulations are valuable, but without real data it is difficult to calibrate them. Many of the platform-specific errors only show up after poring through a lot of data.

There is also the issue of variability. Personally, I have observed on Ion Torrent that we have had some excellent data sets, and we have had some horrid ones. I've also noted that the worst of the worst have been on my least diverse amplicon sets, but the N is too small to draw a conclusion.


Endre Major said...


I have seen the date and got the joke, but I thought that you are half serious. We develop NGS related some software and spent a lot of time with quality control. Our experience is that it is hard to find decent software for some minor or newer platforms. So people who are about to buy an equipment have a hard time to make a choice. There is a lot going on about Ion Torrent but I think software support is not that great. Their variant call is neither perfect nor easily downloadable.