Big data is a hot topic right now--insert rant about new names for old ideas here--and that wave is finally breaking on the clinical lab shores. So the time is right for my "Big Data and Lab" manifesto. (Or at least an observation born of decades of experience.)
Big data has two jobs: archiving and analyzing. Both involve data on computers. There, I claim, the similarity ends. While it is tempting to try to kill both of these birds with a single stone, I assert that this is a terrible idea.
Specifically, I find that in order to archive data effectively, I need a free-form, attribute rich environment which accommodates evolving lab practice without losing the old or failing to capture the new. But in order to analyze effectively, I need a rigid, highly optimized and targeted environment, preferably with only the attributes I am analyzing and ideally with those attributes expressed in an easily selected and compared way.
In other words, I find that any environment rich enough to be a long-term archive is unwieldy for analysis and any environment optimized for analysis is very bad at holding all possible attributes.
Specifically, I have seen what happens to inflexible environments when a new LIS comes in, or a second new LIS and the programmers are struggling to fit new data into an old data model which was a reworking of an older data model. It ain't pretty--or easy to process or fast to process. I have also seen what happens when people, especially people without vast institutional knowledge, try to report on a flexible format with three different kinds of data in it. They get answers, but those answers are often missing entire classes of answers. "They used to code that HOW?" is a conversation I have had far too many times.
Yes, I am aware of Mongo & co and of the rise (and stalling) of XML databases and of the many environments which claim to be able to both. I have seen them all, I have tried them all and I have not changed my views.
So I use a two pronged approach: acquire and store the data in as free-form a manner as possible--structured text, such as raw HL7 or XML are great at this--and then extract what I need for any given analysis into a tradition (usually relational) database on which standard reporting tools work well and work quickly.
The biggest clinical lab-specific issue I find is the rise of complex results, results which are actually chunks of text with a clinical impression in them. For such results, we ended up cheating: tagging the results with keywords for later reporting and asking the clinicians to create simple codes to summarize the text.
I am eager to be proved wrong, because the two pronged approach is kind of a pain in the neck. But so far, I have seen no magic bullet that stands the test of time: either the reportable format is too rigid to hold evolving lab data or the flexible format is too slow to actually handle years and years of data.
If you feel that you have a better idea, I would love to hear it so leave a comment.
Saturday, June 6, 2015
Saturday, February 28, 2015
Too Many Hats
The clinical lab has not been immune to the budget pressure and management thrashing around that has been endemic in American businesses for the past decade--rapid reorganization, loss of headcount, etc.
Working in the clinical lab space, we have seen jobs eliminated even as millions of dollars are reallocated to information technology: a new LIS or a new LIMS or even a new HIS with an LIS add-on.
For the most part, this new tech has not delivered on its ambitious promises of greater productivity so we have seen Lab forced to handle more and more complex orders with fewer people. Not surprising, then, that we also see an apparent decline in professionalism and procedure.
I do not think that that answer is that Lab people are losing capability. Instead, I think that the answer is this: too few people leads to wearing too many hats. Wearing too many hats leads to doing jobs with too little attention and for which one has too little aptitude or training.
Working in the clinical lab space, we have seen jobs eliminated even as millions of dollars are reallocated to information technology: a new LIS or a new LIMS or even a new HIS with an LIS add-on.For the most part, this new tech has not delivered on its ambitious promises of greater productivity so we have seen Lab forced to handle more and more complex orders with fewer people. Not surprising, then, that we also see an apparent decline in professionalism and procedure.
This puzzled me for a long time: why do the Labs with whom we interact seem to be getting worse and worse at their jobs? If you, as Lab management, are going to have too few people, at least keep the good ones!
Things have declined so much that recently we had to revise our project management planning to include doing lots of what we consider to be the client's work for them: basic validation, vendor relations, systems integration, etc.
Why are we being forced to do so much of this? (If we don't do it, it goes undone, sometimes with serious consequences which are supposedly our fault.) Why are clients who are trying to save money happy to pay us to do this work that once they happily did for themselves?
So I think that the terrible Sys Admin I recently had to suffer with is actually an overburdened but excellent bench tech.
Similarly, that incompetent IT manager from earlier this year is a solid tech supervisor trying to do IT management as an afterthought to supervising a lab.
I bet that the hopeless systems integrator whose incompetence forced me to rewrite some code three times is a more correctly termed a lab director whose PhD in genetics did not cover systems integration or software specification or data flow analysis.
Similarly, that incompetent IT manager from earlier this year is a solid tech supervisor trying to do IT management as an afterthought to supervising a lab.
I bet that the hopeless systems integrator whose incompetence forced me to rewrite some code three times is a more correctly termed a lab director whose PhD in genetics did not cover systems integration or software specification or data flow analysis.
Lab is more than down-sized: it has been wrong-sized and it is overly invested in IT. At least that is how it looks from my IT keyboard. So I will try not to shoot the messenger--or the over-taxed Lab person doing someone else's job. After all, I would make a pretty bad bench tech or supervisor or director: God willing, no one will ever ask me to try.
Thursday, January 8, 2015
Too Many Options = Project Problems
I am finishing up my first instrument interface in quite a while. It did not go as smoothly as it should have and the culprit was complexity, aided by the distraction of options.
Instrument Interfacing Then and Now
In days of yore, automated analyzers could be coaxed to spew out ASTM messages out of a simple serial port and you could capture that ASTM for what little processing was needed and then send that data to the LIS, the lab's computer system.
Then came network-capable instruments and HL7, which was an improvement in many ways but much more complicated. (More complexity: sometimes there was ASTM over network and rarely HL7 over serial.)
The LIMS concept took off: a kind of middleman between the instruments and the LIS. The lines became blurry: you could verify on the instrument, on the LIMS or in the LIS. Orders can from the HIS via either the LIS or the LIMS.
With each wave of change, I shrugged and did whatever was needed. Sometimes it was easier; sometimes it was harder. Often things improved for users. There was certainly more to debug when things went wrong.
New Instrument Interface
In theory, after a few years of progress, this instrument interface should have been easier than previous efforts, because automated analyzers are so much smarter than they used to be. Better yet, I controlled the LIS end in this case because I wrote it and I maintain it.
And it was easier, each of the three times I wrote it. Overall, the interfacing was not really any easier than in days gone by.
Why did we have to write the interface three times?
Conclusion
More options do not always make projects smoother or better--or even make consumers happy (see this Fast Company article for details).
As the Lab IT landscape becomes more cluttered, I find that projects are harder to define, longer to implement and more expensive to execute. I would like to say that the rise of the giant Commercial Off The Shelf (COTS) software is fixing this problem, but my experience is that COTS is making it worse.
So the next time your IT folks say "it isn't that simple" press them for details but be prepared to believe them.
Instrument Interfacing Then and Now
In days of yore, automated analyzers could be coaxed to spew out ASTM messages out of a simple serial port and you could capture that ASTM for what little processing was needed and then send that data to the LIS, the lab's computer system.
Then came network-capable instruments and HL7, which was an improvement in many ways but much more complicated. (More complexity: sometimes there was ASTM over network and rarely HL7 over serial.)
The LIMS concept took off: a kind of middleman between the instruments and the LIS. The lines became blurry: you could verify on the instrument, on the LIMS or in the LIS. Orders can from the HIS via either the LIS or the LIMS.
With each wave of change, I shrugged and did whatever was needed. Sometimes it was easier; sometimes it was harder. Often things improved for users. There was certainly more to debug when things went wrong.
New Instrument Interface
In theory, after a few years of progress, this instrument interface should have been easier than previous efforts, because automated analyzers are so much smarter than they used to be. Better yet, I controlled the LIS end in this case because I wrote it and I maintain it.
And it was easier, each of the three times I wrote it. Overall, the interfacing was not really any easier than in days gone by.
Why did we have to write the interface three times?
Version1: Real-time Instrument to LIS
We went through the pain of network configuration (not least punching a hole in at least one firewall) to connect the instrument to the LIS. I wrote the simple HL7 message processor to load the data as it came off the instrument. The HL7 was a little odd, but that was not a big deal. Hurray! Mission accomplished.
Version 2: Real-time LIMS to LIS
Then the users decided that they wanted to use the LIMS that they already had, because order and data entry was better on the LIMS than on the instrument. So they connected their instrument to the LIMS and we connected the LIMS to our LIS and I rewrote the HL7 handling to accommodate the variant HL7 produced by the LIMS. Hurray! again.
(Then some data arrived in the instrument HL7 dialect. Whoops! the users were experimenting with not using the LIMS. We didn't mind, did we? Then more data arrived in the LIMS dialect: yes, they were going to use the LIMS. Almost certainly.)
Version 3: Batch LIMS to LIS
Whoops. The users decided that they wanted the data entry on the LIMS, but to do verification on our LIS which implied to them that the results would be batched on the LIS. So I rewrote the network layer to cache the HL7 messages until the users felt that the results constituted a "batch". Hurray, except that now we are over budget and they have changed their deadline to allow the lab to change their procedures.
Conclusion
More options do not always make projects smoother or better--or even make consumers happy (see this Fast Company article for details).
As the Lab IT landscape becomes more cluttered, I find that projects are harder to define, longer to implement and more expensive to execute. I would like to say that the rise of the giant Commercial Off The Shelf (COTS) software is fixing this problem, but my experience is that COTS is making it worse.
So the next time your IT folks say "it isn't that simple" press them for details but be prepared to believe them.
Monday, September 22, 2014
Advice For Pre-processing Lab Orders
Last weekend, in a supposedly social situation, I happened to run into a clinical pathologist and talk turned naturally to clinical lab informatics. (Yes, the rest of the table was pretty uninterested.)
This CP mentioned that his organization is in the early stages of processing clinical lab orders. This is a great idea, but depressingly infrequent. This is a great idea because:
There are some orders which cannot be done (incorrect collection, eg) or will not be done (policy violation, eg) and the sooner you tell the ordering clinician that, the sooner they can do whatever makes sense.
The earlier you flag a doomed order, the cheaper and easier it is to handle it. Why waste precious human attention in the lab if a rule set can do the job so much faster and just as effectively?
I have lots of free advice--worth every penny, as they say--to offer on the tchnical implementation of these kinds of schemes. However, we were not able to cover the implementation advice in a social setting, much to every one else's undoubted relief. So instead I thought that I would pound out this blog post and send him a link.
What exactly are we talking about? Let us consider the usual life-cycle of the lab order:
(HIS=Hospital Information System; EMR=Electronic Medical Record; LIS=Laboratory Information System)
Conceptually, this software is *VERY* similar to the typical auto-verification module which applies rules to the raw lab result to ensure that the result can be reported.
When an order fails the pass the rules, it is cancelled and a comment added "Cancelled per lab policy: {explanation or link to explanation}".
Since a "hard stop" on clinician's practice of medicine makes everybody nervous, we usually start with the encoding lab policy in the rule set: if the lab is not going to the test anyway, there can be no harm in rejecting the order.
Since implementation is never flawless, we start out with "advisory mode': the module flags orders it _would_ cancel, so a lab tech can confirm that the logic is working. It is a good idea to have an advisory period for every new rule and to audit some percentage on an on-going basis.
So the enhanced model looks like this:
"Correctly" probably requires knowing the leading causes of cancellation and tracking causes of cancellation and confirming that as you add rules, you cut down on the manual cancellations.
There it is: how best to roll out this kind of thing, in my experience.
This CP mentioned that his organization is in the early stages of processing clinical lab orders. This is a great idea, but depressingly infrequent. This is a great idea because:
There are some orders which cannot be done (incorrect collection, eg) or will not be done (policy violation, eg) and the sooner you tell the ordering clinician that, the sooner they can do whatever makes sense.
The earlier you flag a doomed order, the cheaper and easier it is to handle it. Why waste precious human attention in the lab if a rule set can do the job so much faster and just as effectively?
I have lots of free advice--worth every penny, as they say--to offer on the tchnical implementation of these kinds of schemes. However, we were not able to cover the implementation advice in a social setting, much to every one else's undoubted relief. So instead I thought that I would pound out this blog post and send him a link.
What exactly are we talking about? Let us consider the usual life-cycle of the lab order:
(HIS=Hospital Information System; EMR=Electronic Medical Record; LIS=Laboratory Information System)
- HIS
- clinician logs into an HIS
- clinician identifies a patient
- clinician specifies one or more lab tests
- clinician submits the order to the LIS
- HIS generates a lab order
- Order Interface
- accepts order from HIS
- sends HIS order to the LIS
- LIS
- LIS receives the order
- specimen is collected
- lab receives the specimen
- specimen is prepped if need be
- assay is run
- result is verified & posts to the LIS
- result leaves the LIS, en route to HIS
- Result Interface
- accepts result from LIS
- sends result to the HIS
- HIS (full circle:
- clinician sees result, is enlightened
Conceptually, this software is *VERY* similar to the typical auto-verification module which applies rules to the raw lab result to ensure that the result can be reported.
When an order fails the pass the rules, it is cancelled and a comment added "Cancelled per lab policy: {explanation or link to explanation}".
Since a "hard stop" on clinician's practice of medicine makes everybody nervous, we usually start with the encoding lab policy in the rule set: if the lab is not going to the test anyway, there can be no harm in rejecting the order.
Since implementation is never flawless, we start out with "advisory mode': the module flags orders it _would_ cancel, so a lab tech can confirm that the logic is working. It is a good idea to have an advisory period for every new rule and to audit some percentage on an on-going basis.
So the enhanced model looks like this:
- HIS
- clinician logs into an HIS
- clinician identifies a patient
- clinician specifies one or more lab tests
- clinician submits the order to the LIS
- HIS generates a lab order
- Order Interface
- accepts order from HIS
- validate order
- order OK: sends HIS order to the LIS
- order failed: cancel with explanation
- LIS
- LIS receives the order
- specimen is collected
- lab receives the specimen
- specimen is prepped if need be
- assay is run
- result is verified & posts to the LIS
- result leaves the LIS, en route to HIS
- Result Interface
- accepts result from LIS
- sends result to the HIS
- HIS (full circle:
- clinician sees result, is enlightened
"Correctly" probably requires knowing the leading causes of cancellation and tracking causes of cancellation and confirming that as you add rules, you cut down on the manual cancellations.
There it is: how best to roll out this kind of thing, in my experience.
Friday, July 11, 2014
The Most Effective Guy In The Room
I am not a fan of the "smartest guy in the room" meme. I feel that it misses the point. In a business setting, all I care about is effectiveness. Most business problems are resolved by insight, experience, social skills, hard work and luck. Smartness is not usually much of a factor.
The point of being in such a room is to fix an issue, address a lack or form a plan of action to achieve something. Who is going to get us there, in a reasonable amount of time with a maximum of team buy-in? That is who I want to identify. That is who I want to run the meeting.
Since I am consultant and since I have been at this for 30 years, I have run into a wide range of styles and manners and approaches in meetings. There is more than one way to be effective, but I have worked with only a handful of people I thought were terrific runners of meetings. What they have in common is the ability to get at the core issue, to elicit suggestions and solutions, to distribute the necessary work and to keep everyone on task. I don't know how smart they were; it didn't really come up. I was too busy being focused on the job at hand.
As a technology guy, I am usually in meeting about technology and usually with people who are not technologists. So what is required is the ability to lay out technology issues in an accessible way, to facilitate group decisions about technology and to follow up in a way that non-technology people can relate to.
Phooey on smart. Who is the most effective person in the room?
The point of being in such a room is to fix an issue, address a lack or form a plan of action to achieve something. Who is going to get us there, in a reasonable amount of time with a maximum of team buy-in? That is who I want to identify. That is who I want to run the meeting.
Since I am consultant and since I have been at this for 30 years, I have run into a wide range of styles and manners and approaches in meetings. There is more than one way to be effective, but I have worked with only a handful of people I thought were terrific runners of meetings. What they have in common is the ability to get at the core issue, to elicit suggestions and solutions, to distribute the necessary work and to keep everyone on task. I don't know how smart they were; it didn't really come up. I was too busy being focused on the job at hand.
As a technology guy, I am usually in meeting about technology and usually with people who are not technologists. So what is required is the ability to lay out technology issues in an accessible way, to facilitate group decisions about technology and to follow up in a way that non-technology people can relate to.
Phooey on smart. Who is the most effective person in the room?
Monday, June 30, 2014
App Retirement Party
Today marks the beginning of retirement for one of my apps--this one, in fact. As its Viking funeral ends in it sinks beneath the surface of the inky black waters of oblivion, I take a few minutes to ponder its useful life and its lessons.
Given its original mission, it was an astounding success: it allowed Outreach to dramatically grow its volume while greatly lowering the processing burden on the hospital lab.
Its secondary goal of paving the way for software in the draw station was also achieved, since rolling out its replacement was a fraction of the pain of introducing software to them in the first place.
Its tertiary goal of providing hard numbers with which to manage the collection operation seemed to be a glorious success, but since the organization did not bother to replace this functionality when they retired this app, perhaps this was as obviously valuable as I thought?
As an exercise in customization to support operating procedures, it also seemed to be a great success, but many of the most popular and most effective customizations are not being re-implemented, so their value was lower than it seems to me or their cost in the new environment is too high. The whole "this is better, because it cost more" movement baffles me and perhaps this is part of that.
The customizations which people most complain about losing are these:
Godspeed, former users: may your manual work-arounds be as painless as possible and your upgrades lead you, eventually, to the level of efficiency you once enjoyed.
Given its original mission, it was an astounding success: it allowed Outreach to dramatically grow its volume while greatly lowering the processing burden on the hospital lab.
Its secondary goal of paving the way for software in the draw station was also achieved, since rolling out its replacement was a fraction of the pain of introducing software to them in the first place.
Its tertiary goal of providing hard numbers with which to manage the collection operation seemed to be a glorious success, but since the organization did not bother to replace this functionality when they retired this app, perhaps this was as obviously valuable as I thought?
As an exercise in customization to support operating procedures, it also seemed to be a great success, but many of the most popular and most effective customizations are not being re-implemented, so their value was lower than it seems to me or their cost in the new environment is too high. The whole "this is better, because it cost more" movement baffles me and perhaps this is part of that.
The customizations which people most complain about losing are these:
- the ability to automatically cancel ordered tests, with an explanation, of those tests are not going to be done anyway per Lab policy
- the ability to automatically change an order for an obsolete test with an order for its replacement, with documentation, if an equivalent replacement has been defined
- the ability to handle clinical trials and research draws, to ensure anonymity and proper billing
Godspeed, former users: may your manual work-arounds be as painless as possible and your upgrades lead you, eventually, to the level of efficiency you once enjoyed.
Thursday, June 26, 2014
Still No Silver Bullet
I recently got re-acquainted with Ruby-on-Rails and this made me think of software building tools in general, which reminded me that I like to rant about how badly we creators of software classify, describe and choose our tools when confronted by a new task. This is that rant.
We do a horrible job of assessing our options before we start, explaining our choices to our colleagues (and bosses and clients), reacting to feedback as the project progresses.
Once upon a time, shortly after I got out of college, someone wrote an excellent paper on a timely topic: software programmer productivity. The time was 1986, the author was Fred Brooks and the paper was "No Silver Bullet — Essence and Accidents of Software Engineering".
(For the Cliff Notes version, turn to Wikipedia: http://en.wikipedia.org/wiki/No_Silver_Bullet. The paper is worth a read on its own, but who has time?)
The summary in the Wikipedia page is very good, and I will be stealing from it liberally in sum future rant about how badly managers tend to manage programmers and how badly programmers support their management. But you should read it now before the rant really gets going.
This is my banal insight: so little of the fundementals of creating software has changed in the intervening decades since 1986 that sometimes I want to cry. So much has changed about the incidentals that sometimes I want to cry. After 30+ years in the business, I was hoping for more improvement, not more novelty.
And boy, do we have novelty. We love novelty. We worship novelty. Take web UIs as an easy example: we have so many ways to make web pages, my head spins.
I do not have a problem with more than one way to do something, especially if the ways have pros and cons. I am not an operating system bigot: I used VM/CMS for large batch jobs. I used VMS for file-oriented processing. I used Unix for distributed systems. I used MS-DOS for desktop apps. I used OS/2 and then Windows for snappier desktop apps. Each had its pros and cons and its appropriate uses.
The same was true for computer languages: I have used BASIC for matrix math, I have used C for file I/O, I have used Lisp for list processing, I have used PL/1 because IBM required that I use it.
Then somehow this idea of appropriateness faded away from computing: desktop=Windows, server=Unix. Then server=Windows Server for some, Unix for others. C++ or VB as the one-and-only programming language. Then other contenders for the one-and-only programming language.
We all understand that hammers are good for driving nails and screw drivers are good for driving screws. It is clear that screws are better connectors in some situations and nails in others. Who would hire a carpenter who only used one or the other?
But we hire "Drupal Shops" and "C/Windows Shops." We hire "Unix guys" or "Windows guys." We pretend that there is a single best tool for all jobs--and then argue pointlessly about which tool that might be.
Consider this statement from the creator of the Ruby programming language: (full text here: http://en.wikipedia.org/wiki/Ruby_%28programming_language%29)
A single tool to help every programmer in the world to be productive and happy? To me, that seems insane and it seems to reveal a worldview which I cannot support: the idea that there is a single best tool for all people for all problems in all environments. What hogwash.
I applaud the goal of creating good tools and making programmers in general more productive but I reject the notion that this job can every be done with a single stroke.
We do a horrible job of assessing our options before we start, explaining our choices to our colleagues (and bosses and clients), reacting to feedback as the project progresses.
Once upon a time, shortly after I got out of college, someone wrote an excellent paper on a timely topic: software programmer productivity. The time was 1986, the author was Fred Brooks and the paper was "No Silver Bullet — Essence and Accidents of Software Engineering".
(For the Cliff Notes version, turn to Wikipedia: http://en.wikipedia.org/wiki/No_Silver_Bullet. The paper is worth a read on its own, but who has time?)
The summary in the Wikipedia page is very good, and I will be stealing from it liberally in sum future rant about how badly managers tend to manage programmers and how badly programmers support their management. But you should read it now before the rant really gets going.
This is my banal insight: so little of the fundementals of creating software has changed in the intervening decades since 1986 that sometimes I want to cry. So much has changed about the incidentals that sometimes I want to cry. After 30+ years in the business, I was hoping for more improvement, not more novelty.
And boy, do we have novelty. We love novelty. We worship novelty. Take web UIs as an easy example: we have so many ways to make web pages, my head spins.
- Want HTML with embedded code? Try PHP!
- Want code with embedded HTML? Use Perl/CGI!
- Want an abstract environment which separates HTML and code nearly completely? Ruby-on-Rails is for you.
- How about an end-to-end integrated code & HTML environment? Microsoft Visual Studio is just the thing.
- Want to try to side-step HTML and have some portability? Java, in theory, can do that.
- Want to develop HTML with code woven into it? Why, Javascript was created just for this purpose.
I do not have a problem with more than one way to do something, especially if the ways have pros and cons. I am not an operating system bigot: I used VM/CMS for large batch jobs. I used VMS for file-oriented processing. I used Unix for distributed systems. I used MS-DOS for desktop apps. I used OS/2 and then Windows for snappier desktop apps. Each had its pros and cons and its appropriate uses.
The same was true for computer languages: I have used BASIC for matrix math, I have used C for file I/O, I have used Lisp for list processing, I have used PL/1 because IBM required that I use it.
Then somehow this idea of appropriateness faded away from computing: desktop=Windows, server=Unix. Then server=Windows Server for some, Unix for others. C++ or VB as the one-and-only programming language. Then other contenders for the one-and-only programming language.
We all understand that hammers are good for driving nails and screw drivers are good for driving screws. It is clear that screws are better connectors in some situations and nails in others. Who would hire a carpenter who only used one or the other?
But we hire "Drupal Shops" and "C/Windows Shops." We hire "Unix guys" or "Windows guys." We pretend that there is a single best tool for all jobs--and then argue pointlessly about which tool that might be.
Consider this statement from the creator of the Ruby programming language: (full text here: http://en.wikipedia.org/wiki/Ruby_%28programming_language%29)
Matsumoto has said that Ruby is designed for programmer productivity and fun, following the principles of good user interface design.[33] At a Google Tech Talk in 2008 Matsumoto further stated, "I hope to see Ruby help every programmer in the world to be productive, and to enjoy programming, and to be happy. That is the primary purpose of Ruby language."[34] He stresses that systems design needs to emphasize human, rather than computer, needs:[35]
A single tool to help every programmer in the world to be productive and happy? To me, that seems insane and it seems to reveal a worldview which I cannot support: the idea that there is a single best tool for all people for all problems in all environments. What hogwash.
I applaud the goal of creating good tools and making programmers in general more productive but I reject the notion that this job can every be done with a single stroke.
Subscribe to:
Posts (Atom)