Too Much Tech, Not Enough Ology

This post is a rant about how the following common ideas combine to make bad IT, especially clinical lab IT:

• Everyone should express their special uniqueness at all times
• Tech is hard, ology (knowledge) is easy, so go with the Tech
• There is always a better way to do it

Alas! I believe that in many clinical lab IT environments, none of these ideas holds true and that the combination of them is toxic.

We Are All Individuals
You may be a special snowflake, all unique and brimming with insight, but engineering is often about doing the standard thing in the standard way, so those who come after you can figure out what you did. Yes, if you produce something truly revolutionary, it MIGHT be worth the overhead of figuring it out, but it might not.

Consider the mighty Knuth-Morris-Pratt algorithm which was a real step forward in text string searching. However, sometimes we don't need a step forward: we need something we can understand and support.The mighty Knuth himself told a story of putting this algorithm into an operating system at Stanford and being proud, as a primarily theoretical guy, to have some working code in production. Except that when he went to see his code in action, someone had removed it and replaced it with a simple search. Because although the KMP search is demonstrably better in many situations, the strings being searched in most human/computer interactions are short, so who cares? Having to maintain code you don't understand, however, is also demonstrably bad. So Knuth had some real sympathy for the sys admin who did this. Engineering is about accuracy and dependability; learn the standard approach and only get creative when creativity is called for (and worth the overhead).

Tech is Hard, Expertise is Easy
Too often I see clients going with a young programmer because (a) they cost less by the hour and (b) the assumption is that newer programmers are "up-to-date" and doing things "the modern way." I suppose this makes sense in contexts where the domain expertise is somehow supplied by someone else, but all too often I see bright-but-ignorant (BBI) programmers implementing in shiny new development environments with their shiny new skills, but producing a dreadful user experience. The only goal in clinical lab IT is to support the lab, to lower mistakes, to raise through-put, to shift drudgery away from humans and onto computers. If you produce a cool new app which does not do that, then you have failed, no matter how cheaply or quickly or slickly you did the work.

I Did It Myself!
Clinical lab IT deals with confidential information. We are supposed to take all reasonable and customary measures to guard the information in our care. Yet we still end up with security problems caused by programmers either "taking a shot" at security, as though it were an interesting hobby, or doing their own version of some standard tool which already exists, has already been validated and which is already being updated and tested by other people. Don't roll your own if you don't have to--security especially but other important as well. You might enjoy the challenge, but this is a job, not a college course.

Conclusion
Pay for experience. Demand that all novelties be justified. Use what is freely and securely available before rolling your own. Stop the current trend toward "upgrades" which make things worse and endless new systems which do not advance the state of the art. Have the tech and the ology.

Lab Automation vs IT Centralization

Over the past decade I have witnessed two trends in clinical lab computing which I think are two sides of the same coin:

• Lab process automation is going down
• IT is centralized, becoming more consolidated and less Lab-specific

 By "Lab process automation" I mean identifying repetitive tasks performed by humans and transferring those tasks to a computer or computers.

By centralized, I mean that the IT which serves the lab is now generally the same IT which serves all other parts of the organization.

I can see the appeal, especially to bean counters, of centralization: more control by execs, economy of scale, etc. But most of the IT groups I encounter are really all about office automation:

• email
• shared files
• shared printers
• remote access

These are all great if you are running a typical office, which is how everything seems to look from most C Suites.

Alas, the clinical lab is far closer in basic structure to a manufacturing plant than to a law office. Typical corporate IT is not good at process automation:

• receiving orders
• receiving specimens (raw material)
• matching up specimens and orders
• doing the assays (processing raw material)
• serving up results (delivering finished product)

At the bench, email support and file-sharing are not very helpful; powerful and speedy instrument interfaces, audit support and throughput analysis are all much more helpful.

But centralized IT is not only oriented away from this kind of business process automation, they are punished for it: why are you spending time and money on lab-specific projects? If you get email to work on your boss's boss's boss's iPhone, you are a hero. If you figure out how to alert bench techs to specimens which need a smear, you are spending too much time in the lab.

Worse, as corporate IT completes its transition into being mostly about vendor management, the idea of doing what the vendors cannot or will not do--plugging the gaps between off-the-shelf products which gaps cause so much of the needless drudgery in lab work--becomes first unthinkable and then impossible.

Farewell, lab process automation: perhaps vendors will some day decide the interoperability is a goal and then you will live again. But I am not betting on it.

Better Never Than Late

I often complain that the typical organizational IT infrastructure is too complicated and often not well-suited for the clinical lab. I am often asked to given an example, but so far my examples of complexity were, themselves, overly complicated and apparently quite boring.

Well, all that changed recently: now I have nifty example of what I mean.

One of our customers has a fabulous print spooling system: many pieces of hardware, many lines of code, all intended to ensure that your precious print job eventually emerges from a printer, no matter what issues may arise with your network. Best of all, you route all printing through it and it works automagically.

The fancy print job spooler is so smart, it can reroute print jobs from off-line printers to equivalent on-line printers. It is so smart it can hold onto a print job for you, "buffer" it, until an appropriate printer comes on-line.

Alas, neither of these features is a good fit for the clinical labs, at least for specimen labels. The ideal specimen label prints promptly and right beside the person who needs it. If the label cannot be printed, then the print job should disappear: the user will have had to use a hand-written label or some other downtime alternative. Printing the label latter is, at best, annoying. At worst, printing the label later is confusing and leads to mis-identified specimens. For this job, better never than late.

With effort, our client disabled the roaming print job feature, so the labels (almost) always print where they are needed. But the buffer cannot be turned off--it is the whole point of the spooler, after all--and so after downtime, the now-unwanted labels suddently come pumping out of the printers and if the draw station happens to be busy, the old labels mingle with the current labels and opportunities for serious errors abound.

Print spoolers are nifty. They serve office workers well. They are a standard part of today's smart IT infrastructure. But they don't serve the clinical lab in any real sense. The clinical lab is not a typical office environment: don't treat it like one.

Big Data and Lab

Big data is a hot topic right now--insert rant about new names for old ideas here--and that wave is finally breaking on the clinical lab shores. So the time is right for my "Big Data and Lab" manifesto. (Or at least an observation born of decades of experience.)

Big data has two jobs: archiving and analyzing. Both involve data on computers. There, I claim, the similarity ends. While it is tempting to try to kill both of these birds with a single stone, I assert that this is a terrible idea.

Specifically, I find that in order to archive data effectively, I need a free-form, attribute rich environment which accommodates evolving lab practice without losing the old or failing to capture the new. But in order to analyze effectively, I need a rigid, highly optimized and targeted environment, preferably with only the attributes I am analyzing and ideally with those attributes expressed in an easily selected and compared way.

In other words, I find that any environment rich enough to be a long-term archive is unwieldy for analysis and any environment optimized for analysis is very bad at holding all possible attributes.

Specifically, I have seen what happens to inflexible environments when a new LIS comes in, or a second new LIS and the programmers are struggling to fit new data into an old data model which was a reworking of an older data model. It ain't pretty--or easy to process or fast to process. I have also seen what happens when people, especially people without vast institutional knowledge, try to report on a flexible format with three different kinds of data in it. They get answers, but those answers are often missing entire classes of answers. "They used to code that HOW?" is a conversation I have had far too many times.

Yes, I am aware of Mongo & co and of the rise (and stalling) of XML databases and of the many environments which claim to be able to both. I have seen them all, I have tried them all and I have not changed my views.

So I use a two pronged approach: acquire and store the data in as free-form a manner as possible--structured text, such as raw HL7 or XML are great at this--and then extract what I need for any given analysis into a tradition (usually relational) database on which standard reporting tools work well and work quickly.

The biggest clinical lab-specific issue I find is the rise of complex results, results which are actually chunks of text with a clinical impression in them. For such results, we ended up cheating: tagging the results with keywords for later reporting and asking the clinicians to create simple codes to summarize the text.

I am eager to be proved wrong, because the two pronged approach is kind of a pain in the neck. But so far, I have seen no magic bullet that stands the test of time: either the reportable format is too rigid to hold evolving lab data or the flexible format is too slow to actually handle years and years of data.

If you feel that you have a better idea, I would love to hear it so leave a comment.

LabTerm--Proof Is the Product, Not the Programming

Today is the final release of our LabTerm unit.

This release finished a rather unsavoury assignment which yielded a rather good result: a custom lab bench computer or, as we say now, "thin client."

The goal was simple: to create a supportable computer which would do all that a lab tech needs to do:

• provide a full-screen terminal to their homegrown HP MPE LIS (!)
• provide rapid access to their on-line Lab Manual
• provide a web browser for the growing number of web-based UIs
• provide a full-screen terminal to their homegrown order handler

In order to accomplish all of these goals, we decided on the following implementation:

1. a Linux image to run on old (discarded!) former Windows 95 boxes
2. a simple X-windows windows manager, "Rat Poison" (kill the mouse)
3. a custom app to provide a full-screen UI to their Lab Man data
4. a customized version of the MiniCom terminal emulator
1. support for MPE line-handling
2. split-screen support to share the screen (Lab Man or order handler)
3. scroll-back buffer
4. copy-and-paste
5. screen history for each session
5.  automatic updating for all these components, including the Lab Man
6. remote viewing of terminal session history
1. this turned out to be fantastic for providing LIS support

I may have done the last dumb ASCII terminal mankind will ever know, and God knows we didn't break any new ground here, but we produced a product that met the customer's needs. By keeping our eyes on that prize, we succeeded even without the cool factor. And support old hardware meant that we could take cast-off computers from within the organization, so we actually went live with enough hardware to do the job properly.

Engineering can be beautiful, but life often demands ugly and our motto is "beauty is in the eye of the user--not the engineer."

Sysmex CA-500 Automated Analyzer Interface

Today I start another instrument interface. Now the novelty has worn off. I just want to get this puppy up and running.

Still loving the client's homegrown interface framework, but the requirements for these interfaces are pretty strict and the instruments are not very featureful in the output department. These interfaces are a bit of a grind, although it is gratifying to work with users who know exactly what they need and who test so thoroughly. It is also gratifying to be filling such a concrete need.

Cell-Dyn 4000 Automated Analyzer Interface

Today I started work on another automated analyzer interface, this time for the CD4000. I expected that instruments in the same series would be similar, but I was wrong: much like consumer electronics, the individual instruments seem to differ greatly from model to model.

I am still using the same protocol and still using the same very good framework from the client. But this time I will be adding a feature entirely my own: an automated Audit module to support the new-to-the-client feature of....autoverification! Rather behind the industry curve, the client has decided to adopt autoverification but they want strict monitoring and tight control, which is OK by me.

Cell-Dyn 3200 Automated Analyzer Interface

Today I started work on an instrument interface for Abbott's Cell-Dyn 3200 automated analyzer. Most of the work was done for me by the client, whose staff have developed a very solid framework in which to build instrument interfaces.

Automated analyzers are weird: they are sophisticated in every way but the way in which they handle I/O. This has been an introduction to some concepts for me:

• ASTM E1381/95 a rather outdated, clearly serial protocol rather reminiscent of MS-DOS block-based communication protocols of yesteryear.
• Worklists for analyzers which are derived from LIS orders
• Analyzer flags and how to present them
• Verification UIs

It has also been a pleasure to learn about automated analyzer interfaces from a master,  Steve Wardlaw, MD.