Ew, Gross: Why Good Engineering is Sometimes Bad

I like to think of myself as a software architect and not a software engineer.

Engineers are often relieved by this, because they feel that I am not rigorous enough to be an engineer. Architects are often dismayed by this, because I don't wear pleated pants or style my hair.

The practical difference is that I feel comfortable in the creative, design end of things, which means that I often get called in when the engineers have failed.

Sadly, I have come to understand that much of the time, this failure is willful: while I may or may not have vision unmatched by most engineers, I certainly have a willingness to change dirty design diapers. The failure of the engineers, if you listen closely, is often really the sound of a young teenager saying "Ew, gross!"

A function for which I am paid, but which I do not particularly enjoy, is Engineer-to-Management translation. "Why do my IT people say X?" I hear. I try to be diplomatic (for me), but more and more often the honest answer is "The obvious solution strikes them as distasteful, so they don't want to do it."

My favourite example of this was a programmer who embraced object-orientatin to an absurd degree: he re-implemented basic math on top of the native facilities. His code defined ten numeroid objects, '0', '1', '2', '3', '4', '5', '6', '7', '8' and '9.' Of course he then had to define "addition," "subtraction," "division" and "multiplication" for these objects--what fun! His program gave the correct answer, but took 24 hours to run. Running with native integer support, the same program took a little under 8 hours. The programmer's response? "You resorted to an orthogonal formalism." That is the most learned-sounding "Ew, gross!" I have ever heard.

Other examples of holding my nose in the service of providing value include providing users with a dumb ASCII terminal long after such things had become uncool and yucky, or providing fax support despite the outdated nature of faxing.

Engineering is rule-based and rewards rule-following and so attracts rule-loving personalities. They tend to find rules comforting in addition to finding rules useful and productive.

The flip side is that, as a group, engineers are bad at value judgements. They are not inclined to break the rules and they certainly are not comfortable with solutions based on rule-breaking.

Alas, the real world--especially the clinical lab where we are trying to get it right very time--sometimes does not co-operate and requires us to pick best of a bad lot. Even worse, sometimes we need to pick any of a bad lot and get on with giving the users what they need and getting the job done.

Worst of all are the cases where the beautiful engineering is invisible to the users. All they see is that version 2 sucks worse than version 1, again. And again, engineering wants a cookie because version 2 has beautiful engineering and none of the rule-bending messiness that make version 1 so useful.


Sometimes engineering needs to get over itself and do what must be done. That is why I don't mind being scoffed at by hardcore engineers: they may be better at math, but I am better at recognizing when a rule is getting in the way of better solution. I know when to say "that is a good rule, that is a useful rule, that rule will keep you from tending to make a certain kind of mistake, but now is the time to break that precious rule."

Why Can You / Can't You Use The Cloud?

FAQ Wednesday is here again. Today's question: what about the Cloud and clinical labs?

This question has two variants:

1. You can't use the Cloud for heath care data, can you--HIPAA, etc?
2. Why can't you use the Cloud for my clinical lab interface?

Can You Use the Cloud?
The first question, which I take to mean, "is it within law and regulation to use the Cloud for PHI," is actually pretty easy to answer: yes. Does HIPAA restrict the options? Yes. Does HIPAA prohibit use of the Cloud? No.

We currently use Amazon Web Services as our Cloud vendor and they claim to be certified and everything. From http://aws.amazon.com/compliance/#case-studies:

HIPAA

AWS enables covered entities and their business associates subject to the U.S. Health Insurance Portability and Accountability Act (HIPAA) to leverage the secure AWS environment to process, maintain, and store protected health information and AWS will be signing business associate agreements with such customers. AWS also offers a HIPAA-focused whitepaper for customers interested in learning more about how they can leverage AWS for the processing and storage of health information. The Creating HIPAA-Compliant Medical Data Applications with AWS whitepaper outlines how companies can use AWS to process systems that facilitate HIPAA and HITECH compliance. For more information on the AWS HIPAA compliance program please contact AWS Sales and Business Development.

But I expect that Google will keep up and this reference implies that they are:

http://www.healthcareinfosecurity.com/google-amazon-adjust-to-hipaa-demands-a-6133

In fact, we are counting on growing acceptance of Cloud implementations in health care, which is why we are currently developing Direct Interfaces.

Why Can't You Use the Cloud?
This is a slightly difference question, which I take to mean "in practical terms, what are the obstacles to Cloud-based interfacing?" The short answer is "the conservative nature of hospital and clinical lab IT culture." This is very linked to why lab interfacing in general is so hard: our industry punishes mistakes and does not reward innovation. So often, doing nothing is rewarded and thus fighting innovation tooth-and-nail is the norm.

(Since this is legal and low overhead and effective, we plan to step around the hospital and clinical lab IT organizations with our new Cloud-based lab connectivity venture, but that is another story.)

XML vs HL7

It is FAQ Wednesday, when I take a FAQ off the pile and address it.

Today's frequently asked question is "why do so many systems use HL7 instead of XML?"

This is a good question with many possible answers, but this is my executive summary: XML is easy for humans to read and HL7 is easy for computers to process.

Medical IT is often short on power and long on functionality, so it is natural to avoid expensive to process and to embrace easy to process, even at the cost of human legibility. In my experience the people who wonder at the lack of XML are not career medical IT professionals.

XML is a mark up language, a structured tagged text format which descends directly from SGML. It was intended as a platform-independant document storage format, but has become a kind of universal data exchange format.

HL7 is a line-oriented, record and field-based text format which is rather reminiscent of serial line-oriented message formats of yore, such as ASTM which was already familiar to clinical lab people from instrument interfaces.

XML makes more-or-less self-documenting "trees" which can be displayed natively by most browsers, or "visualized" with a little javascript magic: http://www.w3schools.com/xml/xml_to_html.asp There are lots of tools for working with XML and storing it.

In theory, XML is fault-intolerant: XML processing is supposed to halt at the first error encountered. This is not very robust, but in theory there should be no errors because you can write a format document type definition (DTD) which will allow people to make sure that they are sending and receiving data in exactly the way that you expect. If the XML document is made using a DTD and parsed with the same DTD, what could go wrong? And whoever created the data used a validator, such as http://www.w3schools.com/xml/xml_validator.asp on it before releasing it, right?

(In practice, I do not see very much strict adherence to document type definitions.)


HL7 makes nice, simple messages which can be easily processed by almost any programming language. I have written HL7 message processors in C, Perl, PHP, and BASIC.

So how do these formats look like side by side? Consider the following two samples:

HL7 Lab Result:
MSH|^~\&|LCS|LCA|LIS|TEST9999|199807311532||ORU^R01|3629|P|2.2
PID|2|2161348462|20809880170|1614614|20809880170^TESTPAT||19760924|M|||^^^^
00000-0000|||||||86427531^^^03|SSN# HERE
ORC|NW|8642753100012^LIS|20809880170^LCS||||||19980727000000|||HAVILAND
OBR|1|8642753100012^LIS|20809880170^LCS|008342^UPPER RESPIRATORY
CULTURE^L|||19980727175800||||||SS#634748641 CH14885 SRC:THROA
SRC:PENI|19980727000000||||||20809880170||19980730041800||BN|F
OBX|1|ST|008342^UPPER RESPIRATORY CULTURE^L||FINALREPORT|||||N|F||| 19980729160500|BN
ORC|NW|8642753100012^LIS|20809880170^LCS||||||19980727000000|||HAVILAND
OBR|2|8642753100012^LIS|20809880170^LCS|997602^.^L|||19980727175800||||G|||
19980727000000||||||20809880170||19980730041800|||F|997602|||008342
OBX|2|CE|997231^RESULT 1^L||M415|||||N|F|||19980729160500|BN
NTE|1|L|MORAXELLA (BRANHAMELLA) CATARRHALIS
NTE|2|L| HEAVY GROWTH
NTE|3|L| BETA LACTAMASE POSITIVE
OBX|3|CE|997232^RESULT 2^L||MR105|||||N|F|||19980729160500|BN
NTE|1|L|ROUTINE RESPIRATORY FLORA

(from http://www.corepointhealth.com/resource-center/hl7-resources/hl7-oru-message)

XML Lab Result:

<element name="lab-test-results">
        <complexType>
            <annotation>
                <documentation>
                    <summary>
                        A series of lab test results.
                    </summary>
                </documentation>
            </annotation>
            <sequence>
                <element name="when" type="d:approx-date-time" minOccurs="0">
                    <annotation>
                        <documentation>
                            <summary>
                                The date and time of the results.
                            </summary>
                        </documentation>
                    </annotation>
                </element>
                <element name="lab-group" type="lab:lab-test-results-group-type" maxOccurs="unbounded">
                    <annotation>
                        <documentation>
                            <summary>
                                    A set of lab results.
                            </summary>
                        </documentation>
                    </annotation>
                </element>
                <element name="ordered-by" type="t:Organization" minOccurs="0">
                    <annotation>
                        <documentation>
                            <summary>
                                    The person or organization that ordered the lab tests.
                            </summary>
                        </documentation>
                    </annotation>
                </element>
            </sequence>
        </complexType>
    </element>

 (from http://social.msdn.microsoft.com/Forums/en-US/5003cf00-de7f-41ec-93a9-c04b14e41837/xml-schema-of-lab-test-results)

Why Is Merging Test Catalogs So Hard?

It is FAQ Wednesday, when I take a FAQ off the pile and address it.

Today's frequently asked question is "why is merging test catalogs so hard?"

This question fits O'Flaherty's first law of applied technology: "It is usually not the technology that takes the time."

This question arises in a number of contexts:

• Lab Manuals across organization (eg hospital systems)
• Lab Manuals across labs (eg main lab and satellite lab)
• Ref lab interfaces (order from one catalog, process from another)
• Historical data archives (showing results from a previous catalog)

The answer is always the same: assumptions and comparisons.

When considering a lab result, there is a huge amount of assumed context:

1. the collection method sometimes matters, sometimes does not;
2. the methodology sometimes matters, sometimes does not;
3. the actual value sometimes matters, sometime only the High / Normal / Low flag;
4. having a basis of comparison to prior results sometimes matters and sometimes does not

Take Vitamin D levels for example: there are a couple of ways to measure it (methodology) which vary in exactly what they measure and how accurately they measure it. If you are a specialist, these differences may be very important. If you are a generalist, these differences may be meaningless to you. If you are trying to provide historical context by evaluating a series of these results, you almost certainly assume that the methodology was constant, which ever methodology was used.

The problem arises when catalog A only has "Vitamin D" and catolog B has "Vitamin D (the cool one)" and "Vitamin D (the usual one)". How do work across the catalogs?

There is always renaming to enforce consistency, but that is not straightforward and it is often politically charged: someone wins and someone loses. Furthermore, changing the name of a test is problematic: should the clinician infer that the assay has changed somehow? If so, how?

There is add new tests to enforce consistency, but is also a qualified success: what if the new test is essentially the same as the old one? How does the clinician know that, how does the software which displays the results know that these two "different" tests are the same lab assay?

Worse, making these decisions requires many different skill sets: clinical, lab and data modelling. So why is merging test catalogs so hard: because there is so much to it and so much of it is not immediately apparent. Hello, Titantic, meet Ice Berg.

O'Flaherty's First Law of Applied Technology

FQA Wednesday again. In lieu of answering a specific question, today I would like to expound upon something someone else said. Not only is this utterance interesting and apropos, it is also already thought of, saving me some time and effort.

"It isn't the technology that takes the time." Douglas J. O'Flaherty (https://twitter.com/DouglasOF)

Doug has an annoying habit of reducing my intricate and entertaining tales of technology deployment woe to this oft-repeated and simple phrase.

(It is oft-repeated because I complain alot about getting real-world solutions out the door, on to the lab floor and working as well as they should. But that is another story.)

What I understand him to mean is this: information technology is dynamic and evolving, but it is relatively straightforward and relatively well-understood. But in order to apply technology to the real-world business processes that would benefit from that technology, a number of generally poorly-managed, poorly-understood and complex tasks have to be accomplished first:

• get input from stakeholders, who are often diverse and out of phase with each other
• build a consensus from that input
• turn that consensus into specifications which can be acted up
• turn that consensus into action items which really are accomplished

Only at that point does the programming start. When mired in the trenches of implementation, I tend to forget all the pain that went into clearing the path for the implementation and instead I tend to focus on the process failures which leave me with bad specs.

As a technology guy who is constantly frustrated by project scheduling, I guess I do need pretty frequent reminding: it isn't the technology that takes the time.

Medical Data Center Reliability

Well, it isn't Wednesday, but this tweet got me thinking about a FAQ:

#HIT $8k/minute outage, but I know hospital staffs whose SLA is 98% (3+ hours/week!!) #HealthIT http://t.co/19zOXRAWAs
— DouglasOF (@DouglasOF) December 4, 2013

The frequently asked question is this: why aren't Medical Data Centers...better? Given that their mission is so important, not just to an organization but to people's lives, why aren't hospitals and labs better at keeping their systems up and running?

The short answer is that they are trying too hard with an odd set of rules and a strange budgeting process.

Trying Too Hard
In my experience, small medical organizations can't afford good technology or good people, but large ones fail as well for the opposite reason: they are trying too hard.

Redundancy is the obvious way to get good up-time (reliability, uninterrupted service, however you like to characterize "running well"). But there are two different kinds of redundancy....

System Level Redundancy which means having more than one solution to a given problem: two independent, synchronized, preferably different systems doing the same job. If System A has a problem, System B takes over.

Component Level Redundancy which means having more than one component in the system doing a particular job. So you only have System A, but it has lots of redundant disks, and storage and whatever else you think you need.

The computers aboard the now-defunct Space Shuttle were an excellent example of System Level Redundancy: five independent computers, all doing the same tasks at the same time. In order for computer to fail on the Shuttle, all five systems would have to die at once. For good measure, they kept tabs on each other, so you had confidence that they were working properly.

Storage Area Networks are an example of Component Level Redundancy: you tell your server that it has a disk attached, but the "disk" is actually a computer which is sending your data to separate disks.

System Level redundancy is expensive (you need multiples of everything) and daunting (you have to maintain multiple systems and keep them in sync somehow.) But it covers all kinds of failures if you do it correctly: power problems, water mains breaking, etc.

Component Level Redundancy is cheaper (you only have extras of the parts you need) and more inviting (the complexity is hidden). But it makes the infrastructure more complicated and it does not help if a non-redundant component fails, eg water floods your primary data center (you have an offsite back up data center, right?).

Worse, if you choose Component Level Redundancy over and over, you end up with a mind-bogglingly complex infrastructure which punishes attempts to change it and works against attempts to debug it. Is the problem in the virtual machine, or the actual machine, or the network, or the SAN, or the app? Who knows? Let the Blame Roulette begin!

So in trying to be super-reliable, frugal and conservative, Medical data centers often up being flakey, expensive and conservative. At least they're conservative.

But I find that it often isn't their fault: senior management has taken charge of the ballooning tech budget and trying to make simple buying choices which are resulting in complicated situations. And complexity rarely leads to clarity or performance or long-term reliability. Just ask Google or Amazon.

UPDATE Dec-30-2013
This article by a DevOps guy gives some interesting perspective on the same phenomenon:

http://redmonk.com/dberkholz/2013/05/03/devops-and-cloud-a-view-from-outside-the-bay-area-bubble/

Why Can't Medical IT Systems Share Data Better?

It is FAQ Wednesday, when I try to get through the most plaintive of cries I encounter in the course of my workday.

Today's question is "why can't medical information systems share data better?"

This is a good question: why in this day and age of Webly interconnectivity are lab results and diagnostic images and calendar appointments and other data not easily accessible?

Specifically, let us consider why the prototypical Hospital Information System (HIS) cannot share data better (more effectively) with the prototypical Laboratory Information System (LIS).

There are two basic ways to share data between computer systems. I will call these two methods "linking" and "transferring." Let us call the system with the data the "server" and the system which wishes to display data from the server "the  client."

Linking is pretty easy and pretty useful: The client opens a window, sends a query to the server and the server replies with the data.

Transferring is more involved: the client gets data from the server, parses that data and loads that data into the client's own database where that data can found and used by the client's own software.

Linking is easier and real-time, but does not lend itself to a consistent look-and-feel. Transferring is harder and often done in batch mode, but it does lend itself to consistency and the data is available "natively" on the client, ie in more ways.

Since this is healthcare data, we have issues of privacy, authentication, display rules and all the requirements of HIPAA. This makes using Web technology, which is inherently insecure and open, a bit tricky. It also makes the transfer option more attractive: authentication across information systems is a pain and access logging across information systems, especially ones from different vendors, downright difficult.

However, transferring is more overhead to implement, more overhead to maintain and requires actually settling questions of data model mismatch.

We try to support linking when we can, but we often end up having to support someone else's authentication model and someone else's data model and someone else's design aesthetic. That's rather alot of flexibility which is why most bigger vendors won't go that route.

So here we are: most of the time the only data shared between systems is data everyone agrees must be shared. Data sharing tends to lag behind other developments and other requirements. It isn't better because it isn't easy to do.

Why Are Lab Interfaces So Hard?

Taking a frequently asked question off the pile for today's post: "why are Lab Interfaces so hard?" Variants include "why do they take so long to get up and running?" and "why are they so expensive?" and "why don't they work better?"

Here is the most common form of clinical lab interface I encounter:

• Secure network connection: TCP/IP over a Cisco VPN
• Bi-directional: orders from client to lab, results from lab to client
• client has a TCP/IP client for sending orders to the lab
• lab has a TCP/IP server for receiving orders from the client
• client has a TCP/IP server for receiving results
• lab has a TCP/IP client to sending results
• HL7: perfect it is not, but it is common and well-understood
• Assay ID translation: someone has to map the client's test codes to the lab's, or vice-versa

None of this is difficult, technology-wise. I have written such interfaces from scratch in about 8 working hours. Provided that the specs are good, the set up and and programming is more tedious than challenging.

(The specs include the mapping of test codes across menus, by the way. Why that mapping often gets left as an exercise to the programmer I will never understand.)

It is becoming increasingly common to set a target of 24 hours for bringing up such an interface: if you have a working interface framework and a reasonably seasoned team, this should be quite doable.

So-called Direct Interfaces are becoming more talked about. I have yet to see one deployed in the real world in my consulting practice, but I like the idea. In theory this would cut do the network configuration time, but I suspect that the added x509 certificate wrangling will often more than make up for it.

So why does it take days or weeks or even months to get an interface up and running? In my experience, the roots causes are two: bureaucracy and misaligned agendas.

The bureaucracy means that I have spent the majority of my time on projects like this walking from cube to cube, paperwork in hand, looking for permission to connect to another organization. Or hours on the phone, chasing up-to-date, accurate specs and precise explanations of terms like "alternate patient ID".

The misaligned agendas make pulling together in the same direction difficult:

1. The client's lab wants that labor-saving, time-saving, money-saving interface.
2. The client IT department does not want to expose a chink in their armour, which chink will likely never get closed and which chink will be used by an interface which must run 24/7 and always be up.

So what IT hears is "buy a Cisco VPN license; pray that our Cisco VPN appliance has a spare connection; punch a hole in the firewall, but don't botch it leaving us open to hackers; assume responsibility for monitoring the health of that VPN tunnel."

To many IT ears, this sounds like a lot of work, some real money and an on-going obligation with little upside and huge potential downsides. So the network folks become what my firm calls "stoppable"--any hitch in the plan brings them to a halt. Who knows? If they delay long enough, maybe the project will go away....

So why is it so hard, take so long, often have never-resolved issues? Because navigating bureaucracy is exhausting and overcoming organizational inertia takes a long time.