Perl: The Bad Habit I Can't Quite Kick

TULIP is a new assembler for long, error-rich reads such as from nanopore. I was a bit stunned to see that TULIP is written in Perl; I was starting to wonder how many holdouts like me there were. Which led to this exchange on Twitter

@hans_j_jansen @github as someone who can't quite kick the habit, I both applaud&grimace with your use of Perl for leading edge bfx

— Keith Robison (@OmicsOmicsBlog) January 23, 2017

@hans_j_jansen @OmicsOmicsBlog @github if it ain't broken... #perlforever

— Roger Pettett (@zerojinx) January 24, 2017

The Problem with Perl

Well, to me Perl is a bit broken.  There's always been the sheer catastrophic mess of its design, or lack thereof.  My oldest brother, whose opinions I value highly, simply groans that Perl is "pure bad taste".  Of course, taste is, well, a matter of taste, but Perl just doesn't have a well-designed feel to it.  

What happened Perl? You were the future once! https://t.co/2sQ9tiy6Xx

— Keith Bradnam (@kbradnam) January 26, 2017

What's serious though is the rather frequent discovery that Perl libraries are simply outdated, broken or non-existent.  For example, I switched over to Python for the head end of my Twitter extractor, as the Perl library wouldn't work for me.  If I remember correctly, it was a security model behind or so. 

 Similarly, back in summer 2014 I needed a tool to extract FASTA data from MinION FAST5 files.  No tool yet existed within the community.  But again a library failure; my memory is foggy now but I think the Perl HDF5 library couldn't handle some key features.  Either that or it crashed.  So I tested out Julia instead, which worked and I released the first tool for the task within the nanopore community.  I later let that lapse; Nick Loman's Poretools and Mick Watson's poRe are both excellent and I let them manage the headache of supporting an evolving format.

There's more.  I tried out the PDB library back in 2014; a key function required a trivial fix.  Trivial, but still non-functional as delivered.  There was some library I looked into a week or so ago that was just a pilot, but not updated since 2011.  Googling "FASTG Perl" appears to be a dry hole.  Etc.  It's just too common and too frustrating to frequently discover that newer formats aren't available in Perl.  Yes, I could be the one to write them, but those sorts of initiatives usually get dragged down by "gotta get work done now" issues.  Much easier to find workarounds, even when they are ugly.  I could have done that with FAST5; as someone pointed out on Twitter when I griped about this I could  have just piped the output of the command line HDF5 dumper and read that into Perl.  But those sorts of solutions start getting frustrating and slow, though I use them frequently.

Then there's the incomplete stuff that I'm just not up to working around.  Actors are an interesting way to manage multithreading; design your objects correctly and just throw them from actor to actor.  Still not simple, but better than dealing with mutexs and semaphores and such.  I played with them in Scala back at Infinity. There is a Perl lightweight threading library to implement actors, but it comes with warnings that it isn't a final version but a research project.  The last copyright date is 2011, suggesting that no further versions are pending.  That doesn't breed confidence.

I almost vowed to bail on Perl when I joined Starfleet, but the press of getting stuff done quickly wore down my resolve.  That was 2011 and Perl's stagnation wasn't quite so clear. A decision I regret deeply, as now I have a ton of working code I rely on that I sure am not going to port to another language, so any switch wouldn't be going cold turkey. Plus, that's not even remotely realistic; there's always those handy one-liners.

There's also the problem that the language is burned into my brain; I've been using it for two decades now.  If someone presents a problem, my mind starts spinning code.  When programs go south, I often have intuition about what went wrong. I can read between the lines on error messages, often quickly getting to the problem.

A skeptic could quickly knock down several of those.  I've gotten fluent in other languages, sometimes very quickly.  I learned C# in no time at Codon.  Better designed languages would set fewer traps for me, meaning fewer bugs and fewer error messages.  I may be getting a little gray around the snout, but I'd like to think I can still learn some new tricks. 

Programming language discussions are apt to melt down into pointless flaming.  I guess since I'm dissing Perl and saying my complaints about some others, that might be expected, but I hope any comments will tend more towards light than heat.  Hope springs eternal, right?

Perl6 - Bah!

But what to switch to?  Don't even ponder contemplating thinking about suggesting Perl6.  First, I've never heard of anyone in bioinformatics using it.  Second, all reports I've seen is that it is an unholy mess, trying to do everything for everyone while staying somehow backwards compatible.  Most importantly, it wouldn't solve the library problem.

The worst part of Perl6 is that it sucked away a lot of developer talent from Perl5.  I suspect Perl6's tortured, seemingly unending gestation also drove a lot of developers away, or scared off new recruits.

PHP - Abandon All Hope Ye Who Program Here

Speaking of awful language design, there is a deep pit in hell reserved for PHP.  Down where Satan's wings have adiabatically frozen the water. I had to look through an ex-colleague's PHP code recently, and it simply reinforced my belief that the world would have been better off with some better Perl libraries.  

C# - Riiiiiiight

I really, really liked C#.  Great language.  But academia gives it the leper treatment, probably since it came from Microsoft.  Library problem again; there is a NET.Bio library but skimming it in 10 seconds doesn't give me encouragement. 

Python - Resisting the Obvious

Python is the obvious choice, and perhaps I resist it because its too obvious.  I've dabbled around in Python, first at Codon Devices, where one developer used it exclusively.  Codon was a real menagerie, with one segment of code in Python, another whole swath in C# and then some of my stuff in Perl and a touch of R (also me).  They all talked to each other mostly through a relational database, though a few other mechanisms existed (COBRA! We actually used COBRA! I think -- seems to bizarre to be real). Oh, and also via IronPython, a C# to Python interface that I played with.   My talented intern at Infinity used Python, and I could read it just fine.  At Starbase I've tweaked a couple of outside packages, but that Twitter grabber is my main bit of Python and its tiny.  

I think I've gotten over my original negative reaction to the "indents matter!" business, though it still frets me a bit since I so often end up refactoring code by adding loops or removing loops or splitting loops across procedures or other things which would seem to change the indentation level.  Perhaps a good editor helps with that, though to be honest with any new editor I tend to learn the bare minimum (the pinnacle of that is vi/vim -- I know how to get out of it!).

R - Why Can't I Love It?

I've periodically dived into R, even used R graphics in a book chapter written while I was at Millennium.  Dabbled a bit at Codon, Infinity and now with the Federation.  Sometimes that's just cookbook analysis of expression data, but I have written a bit of code.

Somehow it just never catches fire.  I haven't quite caught the Zen of R or something.  Dataframes and I aren't yet best buds.  I can't really describe it properly -- or perhaps I'm too embarrassed that my problems might not be very rational - but R just hasn't (yet?) fit me.

Julia: Not Ready for Bioinformatics Prime Time?

R and Python have huge bioinformatics communities, which argues strongly for them.  Julia is a potential up-and-comer.  Lots of interest in scientific computing fields, that important sibling endorsement (Arch was on the JuliaCon organizing committee for several years), interesting possibilities ahead of it with many serious developers 

When I looked at BioJulia a few years ago, though, the library complement was an issue.  Still, there was an energetic bunch of folks participating in the development message boards. I've looked again and progress has been made, but (again, from a very fast scan) it appears a lot of libraries are missing from BioJulia, such as parsing Genbank. Can't blame them too much for avoiding writing a Genbank parser, an exercise in misery from trying to anticipate all the deviations from an ill-specified standard.   Now in theory Julia can be made to use Python libraries, though I haven't figure that out.  But I don't really want to start with one library set and then switch later on.  That could be camouflaged a bit by writing a bunch of wrapper libraries, but that's still some medicine I'd like to avoid swallowing.

What Else?

I dabbled in Scala back at Infinity, which was interesting and I did get some actors working (as noted above), but again it's not caught fire for bioinformatics.  Let's face it, nearly every language has been tried in bioinformatics and somewhere there is a BioX library for nearly every language X  Except Cobol, thankfully.  My father brought me up to hate three things in life: BASIC, Cobol and Fortran, though he relented when BASIC was the only language available on some early PCs.  But few of these efforts have much there there.  I could see the personal growth advantages of trying out Haskell.  I've never learned a LISP-family language; that's a hole in my development perhaps I should fix. Maybe I should play with some newfangled over the internet sensation like Go.  I did Java twenty years ago; the code just looks ugly to me now.  Too much work to do very little things.  I had no flair for C or C++; managing my own memory is a headache I have no interest in.

NOP

Thus does analysis paralysis set in.  There's always some exciting science to read up on.  There's always science I need to read up on for my job.  Ideally those two overlap, but I won't claim perfection there.  Databases to curate, meetings to attend.  Blogs to write.  Primers to design.  Experiments to design, experiments to analyze.  All the fun chaos that is my bioinformatics life.  So I'll slog it out a while longer with Perl and find those workarounds for missing libraries.

Or maybe I'll switch to the runner-up in some programming language survey for last year, a striver that popped +0.91% in their arbitrary measure.  Yup, the clear up-and-comer is a Practical, Eclectic Rubbish Lister.