Calibration laboratories have traditionally been staffed by highly skilled metrologists performing highly accurate measurements, using laborious, manually operated equipment. To obtain good measurement results required years of experience and thorough knowledge of the measurement systems being used. The metrologists’ skill therefore had a major impact on the reliability of the measurement results produced by the laboratory.
As modern technology has become more accessible and affordable, increasingly, modern calibration laboratories are being equipped with highly automated measurement systems driven by complex software algorithms. This often creates the false impression that these laboratories are superior to the old fashioned laboratories. In addition the temptation is created to employ lower skill level metrologists, who do not have and in fact cannot have an intimate understanding of the measurement systems employed. Often, the impact of these decisions, on the reliability of the measurement results being produced by the laboratory, is never measured.
This paper will take a humorous look at automation in general to demonstrate that it is not necessarily better than manual; especially in calibration laboratories. It will also explore some of the current modern technology strategies being employed by calibration laboratories, their pitfalls and the associated risks regarding the reliability of the measurement results being produced. The author will share some personal experiences and propose some hypotheses, supported by Proficiency Scheme results, motivating why the metrology community should be paying more attention to measuring the impact that modern technology may be having on measurement results being produced by laboratories.
Although the author was unable to source a single definition for “modern technology” from a credible source, a broad Google search revealed the following definition which appealed to him. “Modern technology is the improved product of the application of science”. Whilst probably not a formal definition, in the author’s opinion this definition can easily be applied in the laboratory environment.
For the purposes of this paper therefore, any new application of science, which has led to an improvement, is considered “modern technology”.
Fig. 1: Automated irrigation system activated in the rain.
One could conclude therefore that all modern technology leads to improvement, or can we?
How many times have you seen an irrigation system, wasting litres of water, whilst activated during a rainstorm?
How many of you have been placed on hold on a telephone call, with particularly poor audio quality, waiting for an operator, being told how important your call is to the company and that the call is being recorded for “quality” purposes?
Have you ever received a traffic fine for a caravan having been recorded doing 180 km/h through a speed trap, or an E-toll account for a vehicle that has your vehicle’s registration number but is a Toyota and you drive a BMW?
Has it ever happened to you that after a Windows update installation, none of the Microsoft applications work the same as they did before, or have you ever been amazed at a youngster, using their cell phone as a calculator and then reporting that 5 x 25 = 680?
These are just a few examples of where modern technology has been employed with the objective of improving a service/product and reducing the necessity for human intervention and yet quite obviously, nothing has been improved.
Ironically, it is often the very removal of human intervention that results in the services being worse off – rather than better. In the laboratory environment, more and more control of the measurement process is being taken away from the skilled metrologist by the designers/manufacturers without any consideration of the impact.
With over 30 years of experience in metrology laboratories, the author has experienced both “old technology” and “modern technology” and so is therefore perhaps in a rather unique privileged position to make comparisons. It is this very experience that is raising serious concerns about the negative potential impact modern technology is having on the reliability of the measurement results being produced by present day calibration and testing laboratories.
Modern technologies affecting our daily lives
Whether we like it or not, we are being forced to live with rapidly modernising technology, without any choice in the matter.
Modern cars can no longer be repaired on the roadside. The washing machine at home decides what water temperature is best, how long to wash and even the speed of the spin cycle. Although the average cell phone user probably only uses about 10% of their phone’s capability, they have to pay for it anyway. In our offices we need to log calls on the system, to report the fault that your PC won’t allow you to “log in”!
Whilst the impact of these examples is arguably nothing more than frustration and cost, there are areas where the impact of modern technology could be pivotal between life and death. For example, some of the instruments used in modern hospitals are now so sophisticated, that a nurse spends an hour trying to repair a vacuum seal on a wound to stop an alarm sounding, when in fact the fault is that the pipe has not been properly coupled to the vacuum pump! What if this had been a life support system?
The common thread in these examples, is that there is nothing wrong with the modern technology, it is the implementation that requires attention! Ironically, the more modern the technology, usually the less is known by the user, about how it actually works!
Modern technologies being employed in laboratories
Main reasons for the implementation of modern technology in laboratories
Modern technology is being employed in laboratories for a number of reasons. Let us first discuss some of the technologies and their reasons:-
Automation to reduce the human intervention required to control a measurement process. This can also reduce the level of skill required by laboratory staff to perform highly accurate measurements. In Fig. 2 below, a computer programme will run through an entire sequence of measurements, capturing the results, without any need for human intervention during the measurements.
Fig. 2: Typical automated measurement setup under computer control.
Speed to increase the throughput of calibrations or tests and to perform more measurements in less time. In Fig. 3 below, there is no need to stop the train to perform the measurements. They can be performed while the train is moving past the measurement station.
Fig. 3: Measurements performed at speed.
Data capture to reduce human error in capturing the results from an instrument display or from data transcription. Fig. 4 below shows a current pulse measurement of a few nanoseconds in duration. This could not be captured manually and has to be captured by a data acquisition system.
Fig. 4: Fast rise pulse measurement.
Remote measurement which allows measurement results to be captured from a remote location without having to have human staff on-site. This is also particularly useful when measurements need to be performed in hazardous environments such as ionising radiation. Fig. 5 below shows an electricity consumption meter which performs the measurement of the electricity consumption of a customer and transmits this information back to the relevant authority without any human intervention.
Fig. 5: Remote consumption smartmeter.
Improved control of the measurement process which usually results in a reduced measurement uncertainty and improved accuracy. Fig. 6 shows a typical coordinate measuring machine (CMM), performing highly accurate measurements on a motor vehicle engine block. Performing these measurements under computer control improves consistency and therefore the quality of the measurements.
Fig. 6: CMM performing highly accurate measurements on an engine block.
Software and firmware upgrades can result in instrumentation being kept “current”.
Marketing benefit can result from being able to advertise that the latest technology is being employed in your laboratory – better than the lab around the corner.
Reduced cost can sometimes be realised from buying a modern technology integrated instrument when compared to the older technology approach of a suite of separate discreet instruments. Figs. 7 and 8 below indicate the cost advantage in being able to purchase a single instrument to replace several instruments which used to be required to fulfil the same purpose.
Fig. 7: Old discreet instruments making up a DC voltage system.
Fig. 8: Modern integrated equivalent.
Broad categories of modern technologies implemented in laboratories
In order to understand the potential impact of modern technology in the laboratory, it is useful to broadly categorise the technologies into groups:-
Automation of the measurement or testing process: These are typically automated processes, which vary in their level of sophistication from those which control the entire measurement or testing process, including the coupling of the sample to the system, through to those which only automate a part of the system such as performing repeated measurements/tests and capturing the results.
The degree of human intervention required ranges from only having to setup the software parameters through to having to manage all aspects of the measurement process.
Capturing and manipulation of the measurement results: These systems typically automatically capture/import the measurement results raw data and then perform some calculations in order to arrive at the final answer. The calculations can be as simple as an arithmetic mean and experimental deviation, performed by an Excel spreadsheet, or very complex calculations using powerful mathematical applications and algorithms.
Generation of reports: Although in most laboratories, this function is performed by modern word processor software packages, in some instances, the generation of reports is completely automated using careful integration of exported measurement result data and special report generating software.
Electronic communication tools: It is becoming increasingly common practice to electronically communicate measurement or test results with customers, completely electronically without any hard-copy reports being produced. These processes however still need to take into account the requirements of being “authorised” by an approved technical signatory and protect the integrity of the measurement results reported.
Potential impact of modern technologies on measurement results
Let us now look carefully at each modern technology category and assess the degree to which it can possibly negatively impact the reliability of measurement results:-
Automation of the measurement or testing process
The problem with automation is that it can result in errors being introduced faster than manual systems. Due to the lack of human intervention, the errors often go undetected until it is too late.
There is a temptation for laboratory management to implement highly complex, modern technology, automated systems and then lower skilled staff to run them under the false assumption that this saves money in the long run. What they forget to consider, is the situation when the automated system does not deliver the desired results! Will these errors be detected and if so, would the staff have the skill to correct them timeously before they impact the results reported to the customer?
Often, modern technology systems are automated in order to increase throughput of tests or calibrations. This is usually only beneficial for large volumes of tests or calibrations of exactly the same sample or instrument. This is very seldom the case in a calibration laboratory where the volumes are typically low and the instrument types highly variable. In calibration laboratories, often, just the time taken to load and setup the required software procedure can in fact be longer than it would take an experienced metrologist to perform the calibration in the first place.In addition, highly accurate measurements cannot be hurried, they take time. Therefore, there may be little benefit to be gained from automating a measurement process, using modern technology, which by its very nature takes a long time in any case.
There is always a danger with modern technology automated systems that they can be setup incorrectly by unskilled staff. Examples are software parameters such as filter settings, settling times, sampling rates or the entry of appropriate calibration values or correction factors etc. Unless the staff are skilled to the point where they thoroughly understand the instrumentation and measurement process, there is a very real potential for unintentional measurement errors to be introduced.
“Black box” modern technology measurement systems, where the inner measurement principles and operations cannot be interrogated, make it almost impossible to adequately estimate the measurement uncertainty associated with the automatically generated measurement result. Since it is obligatory to report a measurement uncertainty with every measurement result reported on a calibration certificate, this directly affects the reported measurement result. Reporting a measurement result with an unrealistically optimistic associated measurement uncertainty, is akin to reporting an erroneous measurement result.
Capturing and manipulation of the measurement results
In cases where the measurement raw data is directly captured through a modern technology data acquisition system, and then subsequently processed through some algorithm to produce the final result, it is critically important that the software has been setup correctly at the outset. Not to mention that the software has to be adequately validated.What then happens if the abovementioned system has not been designed in-house by metrologists with adequate skill and who thoroughly understand how it functions, but has been purchased from the instrument manufacturer, and they insist that a “software update” is required? How is the potential impact of this change in software on the measurement results going to be assessed? How will this software update now affect the comparability of historical measurement results, with current measurement results – vital in any metrology laboratory.What will happen to the reliability of the measurement results, should the operating system on the controlling computer be upgraded? Will the data acquisition and processing software still operate correctly?
Is the computer, controlling the data acquisition, calculations and storage of measurement results under the full control of the metrologist, or is it connected to a network of some form or another? Remembering that network connectivity does not have to be by means of a cable (Bluetooth or WiFi for example). What if the reliability of the measurement data could be affected, intentionally or unintentionally, by some network software taking over the control of the controlling computer?
Generation of reports
Whilst the modern technology automated generation of reports has many advantages, such as eliminating human error, standardising on format and many others, it does have some major disadvantages. Here, the solution must be considered in the context of the laboratory type of measurements and volume of work. Whilst it probably makes perfect sense to automate the report generation and approval process in a pathological laboratory testing thousands of samples per day, it is highly unlikely to benefit a calibration laboratory calibrating a few hundred instruments a month with a very wide variety of models and types.In the latter case, the automation is likely to reduce the flexibility of being able to generate tailor made certificates or reports for specific customer technical requirements.
Electronic communication tools
In a modern technological, increasingly fast paced society, instant access to information is a fundamental requirement as is the transfer or sharing of information. For testing and calibration laboratories this poses several challenges. Firstly, in order to meet the ISO/IEC 17025 and SANAS requirements, the unique “signature” of an authorised Technical Signatory has to “appear” on any electronically transmitted measurement results.Secondly, once the measurement results have been electronically communicated, their integrity must remain intact so that their reliability or technical validity is not compromised.The modern technology approach of using a file conversion process to say “pdf” documents is no longer a guarantee that the above requirements can be met. More and more software is available free from the internet which enables pdf documents to be edited. Therefore, significantly more complex security measures are required and even these need to continuously evolve to stay “ahead of the hackers”.
General
There is often a general misconception that modern technology is simply “better” than “old” technology without a detailed feasibility study to base this decision on scientific fact and sound economic reasoning.
A salesman will always try to convince you that the product he/she represents is the latest modern technology and yet it never ceases to amaze the author how when they change employer and therefore the product they represent, their viewpoint regarding which product is superior changes too!
Often, modern technology instrumentation is so fancy, that nobody can provide you any technical information about how it actually performs the measurement. The author is often shocked to find that even the manufacturer in some instances, is unable to answer legitimate technical questions about the inner workings of a “black box” instrument for fear of revealing their “trade secrets”. Furthermore, the technical manuals supplied with modern technology instruments seldom contain any detail of how the instrument works.
Whilst older technology instrumentation was typically designed with a 10 to 20 year lifespan and could be technically supported beyond that, (See Fig. 9 , it is not uncommon for modern technology instrumentation to be “beyond support” in as little as four years!
Fig. 9: Old Textronix 465 oscilloscope.
Case studies of modern technology negatively impacting the reliability of the measurement results
These are all examples of which the author has personal knowledge and are shared in this paper to sensitise laboratory management and staff to the dangers of implementing modern technology without adequately considering the possible consequences.
Case study 1
A computer under a Microsoft Windows operating system was used to control an ac-dc transfer measurement process using custom developed software. The computer was hard-wire connected to the network to facilitate the easy storage of high volumes of measurement raw data.
Unbeknown to the metrologist, the installation of a Windows update was initiated by the IT department, via the network. This modified the sampling timings of the measurements which then directly affected the measurement results. Fortunately, the metrologist was highly skilled and intimately familiar with the software which he had written and not only was the error detected early before impacting the results being reported to customers, but it was immediately corrected.
Lesson learnt: Beware if you do not have total control over the computer which controls the automated measurement system.
Case study 2
A complex Excel spread sheet was developed by the author, to calculate the final measurement results for the calibration of a torque transducer and the associated measurement uncertainties. This spread sheet was thoroughly validated and protected.
The IT department then decided that all computers must be upgraded to the latest version of Excel. Running the spread sheet, developed using a previous version of Excel on the computer now running the latest version of Excel corrupted some of the formulae used in the spread sheet. Fortunately, the corruption was easily detected and then had to be corrected in the new version. This had to be re-validated using the latest version of Excel.
Lesson learnt: Validation of software is only valid for the current version of software.
Case study 3
One of the PT schemes run by the NLA-SA is ILC102, a group of four high accuracy resistors in air. Many of the participants obtain unsatisfactory results for the 1 Ω and 10 Ω resistance values.
As a result of personal experience, the author is of the opinion that this is due to the use, in most calibration laboratories today, of modern technology 8,5 digit digital multimeters. These instruments however, have rather complex zeroing procedures which have to be followed when performing four terminal resistance measurements. If these procedures are not followed carefully, the result is a measurement error at low resistance values.
Lesson learnt: The operation of modern technology instruments requires a thorough understanding of how they work and strict adherence to the manufacturer’s operating instructions.
Case study 4
It is not uncommon in chemistry laboratories to find that modern technology instruments can no longer be technically supported after a five year lifespan, sometimes even less.
Lesson learnt: replacement of these modern technology instruments needs to be budgeted for appropriately. In addition, longer term support guarantees need to be contracted into the purchase of laboratory instrumentation.
Case study 5
The incorrect setup of measurement parameters on modern technology instruments can directly affect the measurement result. If the setting up of these parameters is the responsibility of an unskilled “operator”, the erroneous result may go undetected.
In Fig. 10 a modern digital storage oscilloscope is set to use the incorrect sampling rate and the result is an incorrect frequency value. This is called “aliasing”.
Fig. 10: “Aliasing” on a digital storage oscilloscope.
Lesson learnt: Laboratory staff must have the skills to use the instruments correctly and have a fundamental understanding of the measurement process.
Case study 6
Some laboratories have elected to use modern technology word processing power to automatically enter the date of printing of a report or certificate as the “date of issue”. Sounds like a great idea until a customer requests a copy of the original certificate to replace their lost copy. Since nothing has changed on the certificate, the original “date of issue” remains valid and must NOT be changed and yet the minute a copy is printed the date of issue changes to the current date of printing. If this is not detected, it could impact the reliability of the measurement results reported.
Lesson learnt: Implementing time saving automation must carefully consider all possible impacts on the reliability of the measurement results being reported.
Managing modern technology in a laboratory located in Africa
Almost all modern technology instrumentation designed for use in testing or calibration laboratories, originates from either Europe, Asia or the USA and is designed for use in first world laboratories. As such, many of the design considerations are based on how countries and their laboratories in these regions operate.
The author has significant experience in consulting to laboratories on the African continent and can therefore state with confidence that Africa is a very different place with regards to modern technology:-
mains power is often intermittent and unstable,
there are very long logistic lines to source replacement components or spare parts. Even something as simple as a specialised battery can be difficult if not impossible to source,
Africa is generally a very hot and dusty place – even inside some of the laboratories,
most laboratories are poor and cannot afford the luxury of contracting technical specialists or shipping equipment back to the manufacturers for repairs to be undertaken,
computer hardware is way behind that required to run modern technology measurement systems. Furthermore, if the computer fails, getting it repaired is highly problematic rendering the entire measurement system useless,
training opportunities are in many cases unaffordable since they require travel and accommodation,
in many cases, laboratories in Africa are being equipped through donor funding, with modern technology instrumentation, which cannot be adequately supported locally. This is resulting in either the reliability of measurement results being compromised or a testing or calibration service being short-lived. The author is even aware of one laboratory where the equipment has been supplied by Chinese manufacturers and all the documentation is in Chinese!
In most cases, the level of testing or calibration sophistication required to achieve the desired objective in Africa, is much lower than in first world countries. The author is therefore of the opinion the level of technology needs to take this into consideration. Simpler, older technology, would still be fit for purpose, but would remain reliable under the abovementioned constraints.
Conclusion
There is no doubt that there is a place for modern technology in laboratories. It must however be carefully planned and considered before just being implemented to bring about “improvement”. Modern technology must be thoroughly evaluated to ensure that it can be adequately supported locally and support requirements, in terms of spare part availability, fault finding/repair skills and support lifespan, should be contracted in during the procurement of the instrumentation
Staff employed in laboratories must also have the appropriate skill level to not only operate the modern technology, but also to detect errors and correct them, before they impact the results being reported to the customer.
Contact Eddie Tarnow, NMISA, Tel 012 841-3138, eptarnow@nmisa.org
The post Measuring the impact of modern technology on the reliability of laboratory results – a challenge in the modern laboratory appeared first on EE Publishers.