Notes On Instrument APIs Critiques I Delegated to Claude

In my previous post about the Model Hardware Standard from Anthropic, I mentioned some exploration I've been running on instrument APIs.  Given our Catalyst Agent skills, a model such as Claude can evaluate what gaps exist between a given API and what we would require to integrate into our RAC autonomous laboratory environment.  I've run only a few APIs through, and I'm not going to reference which ones, but I have seen some patterns.  The whole process has been very thought-provoking.

All of us in knowledge industries are being forced to grapple with the impact of advanced machine learning models on our fields.  The models can already do many impressive things, and have also progressed at stunning speeds.  It's amazing how quickly one can get a basic understanding of a new scientific niche or setup analysis pipelines - but of course that is a bit scary for anyone who defined their world around being a specialist in a niche or an analysis pipeline architect.   Or being a blog writer.

As an aside, the last few posts have been run through Gemini before posting, with a prompt asking to flag any sentence fragments.   Such fragments have long plagued me due to being interrupted or having my mind jump between thoughts.  I've appreciated readers pointing them out, but it's certainly not my favorite form of readership metering.  Gemini not only flags what I asked it to, but many other questionable sentences with odd structures, dubious verb application, and other sins.  So far, I've each time used the Gemini as just an alert which I manually fix - or do I?  I can't help but read the suggestions, and maybe sometimes they really are best?  Who wouldn't want a copy editor at their beck-and-call?  But conversely, I don't want to cede creative control to a machine.

In a similar vein, one category of reports from passing instrument APIs past the agent is simple glitches in documentation.  These I would definitely score as being useful to all users.  For example, a key link in one example documentation failed with a 404 error - that's definitely not good.  Sometimes the AI complained that different parts of the same document gave inconsistent answers.  Or critical information locked in an image, with no textual explanation for the benefit of the visually impaired or a computational agent.

A bit of a step up are things such as error codes that don't have longer explanations.  Sometimes you can make an obvious guess, but nobody should be guessing here.  A related one was a complaint that values were returned without the units specified.  There are obvious candidate units, but didn't every science teacher since the dawn of time hammer in you must give your units?  Another along these lines is a complaint that the format for an identifier is not specified.

Another level up would be the AI griping about enumerations which it feels are incomplete.  The AI thinks, based on something it found elsewhere, that the instrument has one set of states but the set of state codes has fewer in it.  

Then there are matters of control as well as poking and prodding.  Does the API even let me ask about the current state?  Can I ask the instrument its model number?  Its software version?  Its serial number?  What are the available commands to control from the outside - can I download all the data types and files a user might see via the instrument's console? Can the API drive uploading a protocol file or labware information or liquid classes to the instrument?  Can the API abort a running process?

Keeping track of runs and data can easily bump up against ambiguities or limitations in mapping protocols to data and so forth.  Is the API limited to accessing only the results of the last run or are all the data files or directories uniquely identified in a predictable manner?  

Instrument families or brands get interesting as well.  I mentioned above the need for querying the model number.  Can I map that model number unambiguously to the differences between the models?  If a company has multiple instrument types, are the APIs harmonized or do they look like each was drawn up without any coordination?

A big one for us - but perhaps not for everyone interested in instrument APIs - is having a sincere estimate of run times.  That's critical information when the scheduler is interleaving different protocols on the system so as to drive protocols through the system as efficiently as possible.  Given the number of variables which might affect this, it should definitely be in the instrument's court to calculate such.  If a run might be of variable length, then a good estimate of the maximum is what is desired.  Our scheduler is dynamic, so should an instrument be freed early it can re-jigger plans to take advantage of that.  

There are of course issues around IT configurations and protocols.  We'd prefer Linux, but some systems are on Windows and use .NET or DLLs.  RESTful access is definitely a plus; the SILA2 standard specifies a different data transport model.  

That's a taste from looking over the reports I sent back to the instrument vendors.  Each time I run this, I have the model generate two reports - a briefer one to send back to the vendor and a more verbose one for internal use that might also describe any considerations that reflect onto our internal plans.  I could have had the model consolidate and summarize all of those reports, but I'm still tending to hold the line on not having AIs write text for me that will go out under my name.

As I've suggested above, I think it can be reasonably argued that many, if not most, of the LLM's constructive criticisms will benefit most if not all users.  Better documentation is just better.  Superior modeling of instrument state through error codes will be useful to all.  But some of these do fall into matters of software architecture or software implementation tastes.  

It also occurred to me that in some sense I am gently imposing a strange sort of standard.  I complained in the prior piece that at this time MHS is an unknown.  I don't know the standard I'm measuring these APIs against - I'm letting the model do the heavy lifting here and only tell me what it doesn't like.  The same opaqueness I kvetched about I'm guilty of performing - though I'm not actually claiming this to be a standard.  Just trying to nudge people, that's all - right??

Of course, with all the generally valuable suggestions the agent is giving that aren't specific to our needs, it does suggest that the use of agents to lint code and documentation has not yet become routine, at least in my sample of companies.

As someone who has written a large amount of code during my graduate career and now nearly three decades in industry, it is sobering that I've not looked once at the method signatures nor do I really care what language the APIs are targeting.  I'm ambivalent whether the function calls from one vendor look anything like those from another vendor.  I'm not going to worry about "where's the Python bindings" or any of that - I can barely hold interest in whether it's RESTful or something else.  I'm indifferent to all of that because my trusted LLM agents will smooth all that over, generating whatever glue code is necessary and certainly will figure out how to install all of these.  If it can't, then that's the feedback I give verbatim.  I have shed a great deal of blood, sweat, and tears over installs over the years and will not miss that.  But what does handing that responsibility over wholesale to the computer mean for my identity as a computational biologist?

Love it or hate it, that is the world we are in now.  We offload many tasks to these powerful agents, but at what point will I run out of things I can't offload?  Am I just a Luddite for not letting Gemini clean up this entry beyond pointing out fragmented sentences?  Should I just accept being a project manager for a bunch of intelligent agents?  Those are the questions I grapple with regularly.