Held in Mauno Koivisto Centre, Turku, Finland 14-15th April 1997
Relates to CEN/TC251/WG5 work item EPAS (Evaluation of Physiological Analysis Systems)
Alpo Värri 2.5.1997
Food and Drug Administration (FDA) in USA has initiated some activity which studies the possibilities of establishing procedures for medical software regulation (including also areas which are not covered by present regulatory practices of the regulation of medical devices containing embedded software). In Europe the authorities have not yet started similar work. The expected growth in telematic health care services and the potential security and safety risk relating to the new services call for discussion of the need of regulation and certification of these services. The purpose of the workshop in Turku was to discuss the scope, means and organisation of the regulatory activity in Europe.
CEN/TC251/WG5 has an accepted work item Evaluation of Physiological Analysis Software (EPAS). The scope of EPAS overlaps with some of the themes of the Turku workshop and therefore a summary of the most relevant discussions of the workshop is brought to the attention of WG5.
As the "information society" makes progress, medical software services will be offered through Internet to an increasing extent and it becomes a big business. This may induce risks in some cases, if the patients begin to use the software themselves and begin to calculate the dosage of their medication, for example. At present sales through Internet are not regulated and the responsibility issue has not been solved.
In future most medical devices will be connected to networks. The borderline between device drivers and other software is getting more and more blurred. Medical devices can be certified but medical software cannot. Software industry could learn from pharmaceutical product validation (Pharmagovigilance vs. "Serverovigilance" where the bugs in software are reported to authorities similarly as the side effects of drugs.)
Software certification is problematic because it takes a long time. Sometimes the bugs in software can be noticed only after a long period of time. Certification is, however, needed by the users of the software, patients and insurance companies. It could also benefit the developers as the certification of their software could boost their sales. The question is should products be registered or regulated. Whatever the case it should be done on an international level. The role of authorities should be assisting the certification not preventing progress.
Comments from the audience:
The speaker spoke mostly about copyright issues of medical software and multimedia content. In Europe software cannot be patented. Copyright is the appropriate protection for software. If a medical database has a copyright, it cannot be used as such without due permission of the holder of the copyright.
There is a problem when the quality of software is to be evaluated as merchandise. If the testing of each (software) product is too complicated, the development process could be certified.
A thorough evaluation of medical informatics product includes the evaluation of the system structure, its function and its impact. The reasons for evaluation are many. One important need is to study if information technology is safe. In principle all expensive technology should be evaluated. Evaluation helps to understand information technology. Legal pressures call for evaluation, too. Sometimes the use of new software saves resources in one place and requires more of them in some other place and the total benefit must be determined.
When the structure of a medical informatics product is evaluated, it means the evaluation of the interface, algorithms and the knowledge of the system. The completeness, correctness and consistency of the knowledge can be evaluated by studying the sources of the knowledge and by letting experts to verify it. Unfortunately the experts cannot always agree. The implementation of the knowledge to the system must be evaluated, too. It must be checked that the implementation is faithful to the knowledge itself, correctly structured and that the symbols have been applied in a consistent manner.
There are methods to validate knowledge bases. Informal testing consists of presenting test cases to the system. Formal evaluation consists of checking the accuracy and wording of the advice the system outputs, the adequacy of explanations, the sensitivity of the system to missing elements, the amount of time the use of the system takes etc. The evaluation of the system as a decision-aid function requires a distinct, large, representative test set. A gold standard to compare to must exist and the criteria for correctness must be clear. Suitable indexes to use are false positive and false negative rate, ROC curves etc. The system could also be compared to existing decision-takers and decision aids. One must remember that symptoms cannot cause diseases and that there must be an indication for every therapy.
It must be assessed whether good advice is enough or not. It must be tested that the system provides results fast enough and in a sufficiently understandable way. If the personnel using the system does not consider it useful or does not act as the system suggests, its cannot be seen as a success. These features are studied in laboratory impact studies which measure user attitudes and the differences between the suggestions of the system and actions taken.
Field impact studies measure the real effect of the system on patient outcome. They require larger resources and careful design in order to avoid the many different possible biases. These include biased population, the learning effects, checklist effect, the Hawthorne effect etc. In conclusion, evaluation is necessary for system dissemination, reveals new scientific questions and thus it is key to progress.
Wyatt has written a chapter of evaluation in medical informatics to the Handbook of Medical Informatics and co-authored the book Evaluation Methods in Medical Informatics with Charles P. Friedman. He could be considered as a potential expert to the EPAS project team if such a PT is established.
In mid-1996 the FDA in USA informed that it wants to discuss the regulation of clinical software. As a consequence a consortium consisting of AMIA, CHISA, CPRI, AHIMA, MLA, and AAHSL made a suggestion to FDA. As a premise the consortium stated that so far there are no reports of extensive damages to patients due to medical software failures.
FDA regulates medical devices and that includes the (embedded) software in the devices. If the device is changed considerably, it requires a new FDA approval. FDA is, however, limited in its resources to practise centralised monitoring of everything happening in the field. FDA cannot control how different devices are connected to each other and how software is developed locally in health care establishments.
The consortium lead by AMIA made a suggestion to FDA about medical software regulation. It suggests that responsibility of medical software is also required in the local level and that health care software producers adopt voluntarily good business practices. FDA should concentrate on systems with highest risks to the patients and pay less attention to other systems. Four categories of systems were defined:
Four classes of regulation were suggested, as well. These were:
A Clinical Software Process Quality (CSPQ) Committee has been established in Vanderbilt University as a model for local committees in the AMIA consortium suggestion. ISO9000 has been partly influencing the set-up of the "business plan" of the CSPQ Committee. It has a list of actions relating to the deployment of new clinical software and it develops ethical rules for the purchase of software.
Comments from the audience:
WHO has a constitutional mandate to develop, establish and promote international standards for food, biological pharmaceutical and similar products. The speaker described the process of approval of pharmaceuticals for medical use. The critical assessment requires tests of efficacy, safety, quality and in some countries also cost of the drug. There are rules for marketing of pharmaceuticals and in the national level laws and sanctions are required. A drug information sheet containing information about the acting substance, pharmacological data, adverse effects, dosage forms, name and address of the responsibles etc. must be provided by the manufacturer. The drug approval process cannot be copied as such to medical software certification but something can be learned from it.
WHO member states have turned to WHO because of drug marketing in Internet. It is illegal to send drugs across the border in EU without prescription of a physician. The speaker presented a personal opinion on how Internet-based marketing should be arranged:
The introduction of telemedicine touches many legal issues such as trade relation, regional authority, privacy etc. Forms of telemedicine are, teleconsulting, teleimaging, direct examination with the patient, telesurgery, for example. Since medical image interpretation is expensive, it is a natural object for international service. The only hindrance to trade here is regulatory. A distinction between services and goods must be made. Services are regulated there where the service takes place and they are regulated by authoritative permissions to give such services. Goods can be regulated as such. The new technology enables the production of services far away and thus causes a regulatory problem. In USA the states regulate medical services but the drugs are regulated nationally. Telemedicine has caused irrational legislation in some states as they want to protect the jobs of their physicians.
Medical software is not very good. Software development is not as advanced as house construction because the correctness of the building can be checked easily from the blueprint. Software documents are usually done afterwards and not with great enthusiasm. Therefore the suggestion is that no pre-marketing approvals are introduced but sanctions are sentenced to producers of software which has caused damage to patients. The process model should be as in boats: close all the leaks you find and keep pumping water out all the time.
The speaker described the terminology of testing (verification - the product is compared to specification, validation - the product is compared to user expectations) and in detail different levels of testing which are:
Certification of software is difficult or impossible when the size of the software is meaningful. The software developing companies don't want to give the source code of the software to third parties. 100 % test coverage requires far too much resources. Black box testing cannot find all errors. The user can (in principle) sue the producer for insufficient testing of its software. If the producer applies quality standards in software production such as ISO 9000, it can show that it is doing its best. The suggestion could be that the software development process is certified instead of the software.
In the speaker's opinion the topic of the workshop is up-to-date and important. He told about the European initiatives in the support of medical informatics research. The goal of the fifth framework program of the European Union is products. National policies are wanted for the implementation of these research results and certification is part of this figure.
The speaker presented the processes which are required to bring a pharmaceutical product into market in Europe. The regulation has been developing piece by piece and the speaker's opinion was that it does not fit directly to medical software regulation. Basically there are three ways to gain the acceptance: the national model, bilateral acceptance between two countries and the EU-wide model. The authorities have deadlines to meet in their acceptance process when the producer has supplied all the necessary documents. Sometimes the authorities require additional preclinical testing. The acceptance process is well-known by the pharmaceutical industry and they know what is expected of them.
The ministry wants to reform the health care process and to apply information technology in streamlining it. Many different kinds of networks will be needed to reach the goal of improving the possibilities of the people. Information society is not only meant to be a marketplace for companies - EU has also stated the goal "citizen first". The whole health care system cannot be changed overnight but different transition paths and middleware is needed between the old and new systems. Health care is sectored and each sector is governed by a law of its own and professional borders. The ministry has set committees for seamless health care service chain, patient data card and patient data security and safety.
Copyright issues are regulated by Universal Copyright Convention, Bern Convention, TRIPS agreement, Rome Convention, WIPO Copyright Treaty and WIPO Performances and Phonograms Treaty (WIPO is currently not yet ratified). The producers of medical multimedia products have to obtain permissions to use all the material in the product from their authors or from the holders of the copyright. Permissions are needed also for public presentation and broadcasting of the material. Database protection has not yet been discussed in the diplomatic congress. Liability and applicable law are still open in WIPO although it covers a lot of topics.
The speaker described a regional project of electronic prescription in south-west Finland. The use of networks requires the solution of different security issues. A smart card is used to recognise different parties of the transactions. The data to be transmitted is encrypted. The developers would like to have an authority which could accept the security solutions to be applied and they would be willing to pay for such a certification.
Wallac develops analytical systems for clinical laboratories. Different quality standards are applied in software development (SDLC, ISO9000, ESA, IEEE...). Adherence to these standards gives a possibility to prove the quality statements used in marketing. Certification of medical software is needed if most customers require it. However, it is costly and it is questionable if it is better than good references and the reputation of the company. Even in the best cases the certification can only cover one version of the software. Therefore the emphasis should be put to the certification of the software development process.
First the speaker presented the organisation of CEN/TC251 and its activities. The users require that the medical information technology products are of high quality, economic, safe, interoperable and obey the laws. There are different options for certification such as product testing, development process evaluation (ISO9003, ITSEC/Common criteria), operational systems accreditation (e.g. BS7799) and CEN/TC251/WG6/PT-12 Security categorisation... The world might not yet be ready for certification of medical software and research is still needed. Phase one is to develop standards to test products against. There are different needs for certification such as safety, security and interoperability. Conformance testing would be important to have. Maybe this could be achieved by third party testing laboratories.
Datex produces medical devices for anaesthesia monitoring and for the intensive care units. The devices contain a large amount of software (in the order of 800 000 lines of source code). Thousands of pages of test plans and results are part of the internal product documentation. When a new SW engineer is hired to the company, it takes six to twelve months before he becomes productive due to the large amount of background and product-related information which he has to study. It is therefore unlikely that a third party could check the software so thoroughly that it could certify that the software behaves correctly. The expenses of such a checking attempt would also be high with respect to the expected benefit. It is also unlikely that a big company can afford the bad press which a severe software malfunction can cause and therefore the company will take all reasonable precautions in order to prevent it.
The process of software development could be certified instead of the software products. One good alternative is ISO9001/EN46001. The medical information the manufacturer has used in the development could also be checked. A risk analysis could be performed to the products. Internal product testing should be encouraged. FDA wants to check all documentation and this might be considered in Europe as well. The buyers should be encouraged to look for quality. Some authority could provide lists of manufacturers which comply to ISO9001 to the web. Products of non-ISO9001 compliant companies should carry a warning label. The EU could support the development of evaluation software which should be made publicly available and the use of which should be voluntary.
Telecom and data networking services are well developed in Finland. Helsinki telephone company has a goal that by the year 2000 a 2 Mbit/s line is available everywhere in the area it operates. The role of the teleoperator is changing from the provider of the line into a joint venture with different value added service providers. The Finnet Group has a Medicinet program in which different telemedicine services are developed. The goal is that the teleoperator will be able to offer the required level of data security. At present the health care record is not standardised to a sufficient degree. There is also a conflict between legislation and the concept of seamless care.
What to register or certify? The registration or certification of data security and the network infrastructure and its management should be performed by some authority. The cost-effectiveness of the telemedicine systems should be evaluated. Common standards for medical data representation and interoperability are important. Services generic to all businesses should be created and health care specific standards should not be made unless absolutely necessary. Socio-economic evaluations of the new telematic health care services are important, too.
Health on Net Foundation aims at supporting health care with the help of the World Wide Web. It has a search engine which studies the web looking for medical information. It attempts to establish a world-wide code of conduct for health information providers in the net in order to avoid the dissemination of hazardous or harmful health information. The public would like to get health information from the net but not so many hospitals or holders of the right knowledge provide it. 76 % of those who responded to an Internet questionnaire made by Health on Net were of the opinion that the quality of the health information in the net should be of better quality. The suggested code of conduct to Internet health information providers lists the following principles:
Health on Net can review a web site on request and if it approves its content, it can give a right to use its approval logo on the web page of the health information provider. Many individuals have given hints to Health on Net about web sites which do not comply to the rules given above. The problem is that the review made by Health on Net is easily outdated by changing the essential part of the contents overnight.
The unfortunate time constraints in the end of the workshop did not allow a thorough discussion of the actions to be taken as a consequence of the presentations of the workshop. Instead Jari Forsström concluded briefly what he saw as the most important results of the workshop. These were:
Concerning registration following ideas remained on top:
The participants of the workshop did not plan a joint action in order to promote the ideas in an organised way. The field was left open for new initiatives.
The scope of evaluation of physiological analysis software and systems appears to be wide and not so well defined. It might be the time to activate the EPAS work item in WG5 at least to the extent of defining what does the mandate of WG5 cover in this topic and what should be left for other groups.
Four papers of the workshop appeared in a Special Section on Certification of Medical Software in International Journal of Medical Informatics, Vol. 47, No. 3, December 1997, published by Elsevier Science, Amsterdam, Holland, e-mail: firstname.lastname@example.org.
Signaalinkäsittelyn laitos, Tampereen teknillinen korkeakoulu, PL 553, 33101 Tampere, Puh. 03-3652575, Fax. 03-3653857, E-mail: email@example.com.