Failure: The Real Secret Sauce of Engineering

I took one swing at Vijay Pande's overly rosy piece on applying engineering methods to biology and medicine and similar minded efforts were published by Ash Jogalekar at Curious Wavefunction and Derek Lowe at In The Pipeline. Perhaps I shouldn't make another go, but it is a new excuse to explore an old fascination of mine.  Pande's subhead was "Billion-dollar bridges rarely fail -- whereas billion-dollar drug failures are routine".  I can't argue that.  Actually, it would seem from an informal search that billion dollar bridges are actually much rarer than billion dollar drug development programs.  Obviously they exist -- I've traversed the new Tappan Zee Bridge which came in over $3B.  On the other hand, a second crossing at perhaps the most notorious spot in bridge engineering history, the Tacoma Narrows, was built earlier in this century for only $0.8B.  What I wish to explore are the failures of bridges and other structures of any cost, as it is the analysis of failures that frequently propels engineering forwards.  That analysis is enabled by the relative simplicity of human engineering and the artifacts it uses and creates.  Conversely, analyzing the failure of new drugs is nothing like that.
I do feel I need to emphasize I'm not trying to bash engineering; remember that I am (perhaps excessively) proud of the engineering exploits of my late father.  His father, who I never met, was also an engineer by training.  As a kid I loved crossing bridges and traversing tunnels.  Suspension bridges are by far my favorites, but I enjoy any good bridge.  When I was very young we would stop to see the progression of a major bridge, the Commodore Barry Bridge over the Delaware River, whenever we took relatives to the airport.

As a kid I was obsessed with trains and checked out essentially every book on them at the public library.  At some point that included stories of U.S. train disasters (alas, the library had no good books on railroads in other countries).  Many of these books noted how later engineering improvements either eliminated possibilities for human error or prevented injuries should accidents occur.  After the MGM Grand tragedy, I went for a book on U.S. fire disasters.  At some point airline crashes were checked out.  Later in life I'd find books on bridge and building failures. And for this essay I did some browsing through Wikipedia to remind myself of details on some notable disasters, particularly some relatively recent ones. 

When structures or vehicles fail spectacularly, particularly when great loss of life is involved, skilled investigators comb through the wreckage and any other evidence.  Major bridge failures are relatively rare, particularly in this country, but they are informative.  So are near misses.  One trigger for this piece was seeing an item about the Millennium Bridge in London, a graceful pedestrian bridge I have walked many times.  When first opened it started swaying side-to-side with alarming strength.  The culprit turned out to be a small sway that would be amplified by the people walking on it; to compensate for the motion (and not fall down) people would shift their weight, which succeeded in pumping more energy into the sway.  Active dampers were added to the bridge to eliminate this.  Similar solutions fixed the problem here in Boston of windows popping out of the 60 story John Hancock Tower when it opened in the early 1970s.  And, of course, similar problems brought down the original Tacoma Narrows Bridge aka Galloping Gertie.

A spectacular and tragic bridge failure just before my birth was the Silver Bridge over the Ohio River.  By combing through the wreckage, investigators found a single eyebar that had a defect in it.  The problem had been exacerbated by poor maintenance and growth in load beyond the original design.  The more recent Minneapolis Bridge failure was also tracked down to a single part, with corroborating visual evidence from prior inspections as well as surveillance camera footage that showed the collapse. That bridge had also been loaded beyond the initial design, both by additional pavement and the presence of construction vehicles.

I've mentioned before I mentor the prospective Eagle Scouts at one of the local troops.  Some of these boys have built bridges for their service projects, and while many are relatively simple causeways a few have been serious pedestrian bridges spanning water navigable with a kayak.  For these boys -- and even some with less involved engineering efforts -- there is one disaster I always bring up because I believe it has a useful message: the Kansas City Hyatt Regency balcony collapse.  It occurred while a large group was dancing on two suspended balconies -- dynamic loads are far more troublesome than static ones. But the key lesson for my junior bridge builders is this: a key factor in the collapse was that the original design was nearly unbuildable.  It called for hanging the balconies from common support rods., requiring the effective threading of the upper balcony onto the rods (yes, scaffolding could work wonders here)  Someone "cleverly" found a more buildable solution -- hang the lower balcony off the upper balcony.  Simpler -- but the attachment points for the upper balcony weren't re-scaled to account for the double load they were carrying.  And that was revealed by reviewing the as-built design documents.

Or consider the two space shuttle losses, events that I followed intensely.  In each case there was a gigantic effort to recover wreckage and then piece as much of it back together.  Those efforts, combined with video footage, ascertained the root causes as well as highlighted the fundamental design flaw in the space shuttle: putting the manned vehicle next to rather than atop the cryogenic rockets.  That position allowed debris from the rockets to strike the wings of the craft, which is what doomed Columbia.  

With a small change in snowfall in the Philadelphia area that day, I would have seen Challenger disintegrate live -- instead I found out shortly after the failure when another teacher came bursting into our chemistry classroom.  Video footage revealed a puff of smoke suggesting a failure of an O-ring on the solid rocket booster, corroborated by later footage showing a jet of flame from that joint.  This led to investigating the temperature effects on the O-ring -- as exemplified by Richard Feynmann's famous/notorious live demonstration (William Rogers, the committee chairman, was reportedly furious for Feynmann going off script).  Further analysis, as hammered on by Edward Tufte, illustrated that not only had the shuttle never launched in such cold conditions, it had never launched in temperatures remotely similar to that condition.  

With a variety of evidence, but more importantly good models of forces in these systems, answers could be obtained.  The contributions of various factors could be analyzed.  Specific human-made parts could be assigned levels of blame.  Clearly this is a very different take on engineering than what Pande portrayed.  While large repeatable experiments are vital, so is analyzing inherently singular events such as disasters.  We can attempt informative models, but those are never the same as the actual structures and vehicles in these accidents.

Now let us switch from engineering failures to clinical failures.  For a source of focus, I'll look at the failure of a project I was involved in: IPI-926, an anticancer agent developed by Infinity Pharmaceuticals. During my time there I was involved in efforts to discover additional pharmacodynamic biomarkers for the program; such markers would be used to measure the degree to which the drug was actually affecting target tissues.  I was also involved in a collaboration to explore the biosynthesis of the starting compound (the academic group for that work did publish a paper much later on)

IPI-926 is a semi-synthetic derivative of the plant natural product cyclopamine, a compound with a colorful backstory I'll save for another time .  Like cyclopamine, it is an inhibitor of smoothened, the receptor for hedgehog ligand.  Targeting hedgehog pathway was a hot topic during the first decade of this century. 

Infinity had progressed the compound through Phase I trials largely using patients with genetic mutations in the hedgehog pathway.  Typically these are in the negative regulator Patched and lead to either horrific skin lesions (basal cell carcinoma) or highly lethal brain tumors (medulloblastomas).  Some patients had Gorlin's Syndrome, which is due to germline mutations activating hedgehog signaling and others had somatic mutations.  Responses were seen in many of the BCC cases -- these often present as grotesque skin ulcers and the treatments led to partial or complete closing of such ulcers.  We also had evidence of in vivo effect from a Phase I study in dogs which dosed dogs prior to amputation to treat osteosarcoma (if the disease has not metastasized this is curative; if it has the poor pup is doomed) -- Infinity received the removed limbs and then tested them with pharmacodynamic markers (alas, I added nothing to the quiver there). 

Three Phase II trials were initiated.  The big one was in pancreatic adenocarcinoma, a disease which I had a personal grudge against -- it claimed an uncle-by-marriage who I was quite fond of.  I've since learned that my undergraduate adviser succumbed to the disease and a good friend is currently afflicted.  It's an awful disease with a very poor outlook.  The Phase II would have two arms -- gemcitabine alone or gem+IP-926 -- and was blinded -- essentially a Phase III design (indeed, if things broke right it might lead to early approval).   Other trials were in chondrosarcoma and myelofibrotic disease.  

One of my big regrets on beaming up from Infinity is that I wouldn't be there for the success of IPI-926. We had great data in genetically engineered models of the disease.  It was an interesting and exciting hypothesis -- that hedgehog signalling from the tumor caused the surrounding tissue, known as the stroma, to help defend the tumor against the standard-of-care chemotherapy agent, gemcitabine.  So IPI-926 would help, as our hypothesis went, enable the drug to get to the tumor.  

Alas, there was no success. The pancreatic trial was stopped early because the treated arm was dying faster than the control arm.  The chondrosarcoma trial ended early due to futility. I'm not sure why the myelofibrosis trial ended, but it's a safe assumption it wasn't going anywhere good.

The crash-and-burn in the pancreatic trial was a shock to all involved; I ran into one former colleague at an event a few days later and he got a complete deer-in-headlights look when I asked what happened.  Nobody expects Phase IIs to run flawlessly, but to have the drug fail so miserably was not anticipated by anyone.

So now the question: how do you analyze such a failure?  When a bridge fails one can look at footage of the failure, at the wrecked parts and at design documents.  One can look for clear structural alterations, sloppy construction or environmental effects.  Even when the failure isn't due to things strictly under human control -- poor accounting for the local geology doomed the Teton Dam -- overall these effects can be modeled with reasonable approximations.

But how could this be applied to the IPI-926 pancreatic trial failure?  Even if you could get biopsies from every patient -- not an easy task given they would likely all be post mortem - -what would you look for?  Was it some specific toxicity in the patient population -- which was the determined caused of prior Infinity Phase III failure in another program?  Was the "opening up the stroma" hypothesis wrong?  Did inhibiting hedgehog pathway trigger some other mechanism in the tumors?  Those are all interesting hypotheses -- but how do you even approach them?  Remember, none of this had shown up in the genetically engineered mouse models.  Nothing had shown up in the xenograft models either.  Nor anything like this in the Phase I human trials -- albeit those were in other tumor types.  And it was only because the trial had two arms -- which the Phase I trials didn't -- that the excess mortality was detected.  Given the usual rapid course of pancreatic trial, deaths in a single arm trial would be uninformative.

This is what real world clinical development looks like.  Tough diseases and unclear models.  I might dream of treating genes and cells as abstract Legos, but the reality isn't anything like that.  And unfortunately there's no short-term hope of having anything like a good model of any patients pancreatic adenocarcinoma. It's not like a bridge or balcony where we can run some elegant math to nail down that failure of this or underdesign of that would lead to catastrophe.  That's all too often why billion dollar drugs fail and why if we want to have successes in drug development we need to accept that many more failures are in our future.  Ultimately, the only reliable test of any drug is to put that drug into the patient population.  We'll always look for ways to gain advantage, but thinking that better modeling and testing will eliminate clinical failure is a pipe dream.