Author Name
Admin_niti
Choose Report Type
Publication Date
Report Upload
Download
(3.11 MB)
vertical
Health & Family Welfare
PDF Text
;
A feasibility study to assess the usability,
usefulness, and adherence to Standard Treatment
Guidelines in Indian healthcare settings with the
use of Elsevier’s “Arezzo®” - a Declarative
Artificial Intelligence based Clinical Decision
Support System and pathway technology
Healthcare Artificial Intelligence Catalyst (HAIC) Pilot Study
Sponsored by NITI Aayog (GOI) & UK-DIT
Principal Investigator Team
Lady Hardinge Medical College and
All India Institute of Medical Sciences,
New Delhi
HAIC Pilot
Project Report
JULY 2021
287887/2021/DM&A
481 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
1
A feasibility study to assess the usability,
usefulness, and adherence to Standard
Treatment Guidelines in Indian healthcare
settings with the use of Elsevier’s “Arezzo®” -
a Declarative Artificial Intelligence based
Clinical Decision Support System and pathway
technology
Healthcare Artificial Intelligence Catalyst (HAIC)
Pilot Study
Project Report
July 2021
Sponsored by
NITI Aayog (GOI) & UK-DIT
Submitted by
Principal Investigator Team
Lady Hardinge Medical College and
All India Institute of Medical Sciences,
New Delhi, India.
287887/2021/DM&A
482 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
2
Table of Contents
List of Tables ............................................................................................................................................................ 4
List of Figures .......................................................................................................................................................... 4
List of Abbreviations .............................................................................................................................................. 5
Acknowledgements ...................................................................................................................................................... 8
1. Key Stakeholders and Project Team ................................................................................................................ 9
2. Introduction ....................................................................................................................................................... 11
2.1. Background ................................................................................................................................................... 11
2.2. Arezzo® ......................................................................................................................................................... 12
2.3. Rationale ........................................................................................................................................................ 13
3. Study Aim and Objectives ............................................................................................................................... 13
4. Hypotheses ......................................................................................................................................................... 14
5. Methodology ...................................................................................................................................................... 14
5.1. Study Phases .................................................................................................................................................. 14
5.2. Target Health Workers ................................................................................................................................ 15
5.3. Constitution of Expert Reference Group ................................................................................................ 15
5.4. Preparatory Field Visit ................................................................................................................................. 15
5.5. Infrastructure and Resources...................................................................................................................... 15
5.6. Selection and transformation of Guidelines ............................................................................................ 15
5.7. Detailed Methodology ................................................................................................................................. 16
5.7.1. Phase I ................................................................................................................................................... 16
5.7.2. Phase II ................................................................................................................................................. 18
5.8. Outcome Measures ...................................................................................................................................... 21
5.8.1. STG Adherence Indicators ................................................................................................................ 22
6. Results ................................................................................................................................................................. 24
6.1.1. Base-line Survey ................................................................................................................................... 24
6.1.2. Phase 1 Results .................................................................................................................................... 24
6.1.3. Phase 2 Results .................................................................................................................................... 26
6.1.3.1. Quantitative Scorecard Surveys Analysis .................................................................................................. 26
6.1.3.2. STG Adherence ................................................................................................................................... 32
6.1.3.3. Qualitative data: Focused Group Discussions ............................................................................... 35
7. Discussion .......................................................................................................................................................... 41
8. Lessons learnt and lateral insights for future studies .................................................................................. 43
9. Conclusion ......................................................................................................................................................... 48
10. References .......................................................................................................................................................... 49
11. Annexures .......................................................................................................................................................... 50
Annexure 1: Arezzo® Validation ....................................................................................................................... 50
287887/2021/DM&A
483 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
3
Annexure 2: Survey Questionnaires ................................................................................................................... 53
Annexure 3: Baseline Survey Results .................................................................................................................. 58
Annexure 4: Phase 1 testing case mix ................................................................................................................ 66
Annexure 5: Interim Survey Results ................................................................................................................... 70
Annexure 6: STG Adherence Secondary Data Analysis on limited journeys .............................................. 72
287887/2021/DM&A
484 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
4
List of Tables
Table 1 Target Health Workers
Table 2 Guidelines selected for the HAIC Pilot
Table 3 HAIC Study Participants
Table 4 HAIC Study Outcome Measures
Table 5 STG Adherence Indicators
Table 6 Phase 1 Reliability Testing Journeys
Table 7 STG Adherence Indicators for Maternal Health
Table 8 STG Adherence Indicators for Child Health
Table 9 Key Stakeholders of the HAIC Study
List of Figures
Figure 1 HAIC Study Gantt Chart
Figure 2 HAIC Study Flow-Chart
Figure 3 Baseline Survey Results
Figure 4 Phase 2 Qualitative Survey Results Summary
Figure 5 Usability Survey Results
Figure 6 Usability Survey Stacked % Results
Figure 7 Usefulness Survey Results
Figure 8 Usefulness Survey Stacked % Results
Figure 9 Overall Satisfaction Survey Results
Figure 10 STG Adherence Trend (Oct-2020 to April 2021)
287887/2021/DM&A
485 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
5
List of Abbreviations
AIDS Acquired immunodeficiency syndrome
AIIMS All India Institute of Medical Sciences
AMRIT Accessible Medical Records via Integrated Technologies
ANC Antenatal Care
ANM Auxiliary Nursing Midwifery
ANOVA Analysis of Variance
API Application Programming Interface
ASHA Accredited Social Health Activist
BP Blood Pressure
CDS Clinical Decision Support
CDSS Clinical Decision Support System
CHC Community Healthcare Centre
CI Confidence Interval
CIG Computer Interpretable Guideline
COVID-19 Novel Coronavirus disease (2019)
CVD Cardiovascular Disease
DBP Diastolic Blood Pressure
DMPA Depot Medroxyprogesterone Acetate
EDD Expected Date of Delivery
EHR Electronic Health Record
ERG Expert Reference Group
ETAT Emergency Triage and Treatment
FBNC Facility Based New-born Care
FGD Focus Group Discussion
FHW Frontline Health Workers
GDM Gestational Diabetes Mellitus
GOI Government of India
HAIC Healthcare Artificial Intelligence Catalyst
Hb Hemoglobin
HBNC Home Based New-born Care
HCW Healthcare Worker
287887/2021/DM&A
486 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
6
HIV Human Immunodeficiency Viruses
HRP High Risk Pregnancy
IFA Iron Folic Acid
IMNCI Integrated Management of New-born& Childhood Illnesses
IMR Infant Mortality Rate
I-NIPI Intensified National Iron Plus initiative
IYCF Infant and Young Child Feeding
JSSK Janani Shishu Suraksha Karyakram
JSY Janani Suraksha Yojana
KMC Kangaroo Mother Care
LBW Low Birth Weight
LHMC Lady Hardinge Medical College
LHV Lady Health Visitor
LMP Last Menstrual Period
LSU Lactation Support Unit
MCP Mother and Child Protection
MMR Maternal Mortality Ratio
MO Medical Officers
MOHFW Ministry of Health & Family Welfare
NCD Non-Communicable Disease
NFHS National Family Health Survey
NHM National Health Mission
NIS National Immunization Schedule
NITI National Institute for Transforming India
OGTT Oral Glucose Tolerance Test
OPD Out-Patient Department
ORS Oral Rehydration Solution
PHC Primary Healthcare Centre
PIH Pregnancy Induced Hypertension
POC Proof of concept
PPIUD Postpartum Intrauterine Contraceptive Device
PSMRI Piramal Swasthya Medical Research Institute
287887/2021/DM&A
487 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
7
ReST Representational State Transfer
Rh Rhesus Factor
RO Research Officers
SAANS Social Awareness and Actions to Neutralize Pneumonia Successfully
SAM Severe Acute Malnutrition
SBP Systolic Blood Pressure
SC Sub-Center
SDG Sustainable Development Goals
SIM Subscriber Identification Module
SN Staff Nurse
STG Standard Treatment Guidelines
TT Tetanus Toxoid
U5MR Under Five Mortality Rate
UK-DIT Department for International Trade, Govt of United Kingdom
UN United Nations
UP Uttar Pradesh
VDRL Venereal Disease Research Laboratory test
VHND Village Health, Sanitation, and Nutrition Day
WHO World Health Organization
287887/2021/DM&A
488 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
8
Acknowledgements
The HAIC Study represents a massive collective effort of several teams who persevered through several
obstacles and successfully completed the study despite several unforeseen challenges, COVID-19 being the
foremost. The willingness of each member to go the extra mile was due to a keen inclination to contribute to
improving health outcomes with the use of advanced technology grounded in knowledge-based systems.
Sincere thanks are due to the National Institution for Transforming India (NITI Aayog), particularly Prof V
K Paul who envisioned the study and invited us to lead it. Our special thanks are due to Ms. Anna Roy, Mr.
Sabyasachi Upadhyaya and the Panel of Experts from NITI who steered this study and provided expert
guidance and facilitation, that was cardinal to the scientific conduct, smooth administration, Government-
level approvals, and overall quality of this study.
The Principal Investigators would like to place on record their appreciation for the Government of the
United Kingdom and the UK- Department of International Trade (DIT) for sponsoring this study and
generating evidence for the feasibility of Artificial Intelligence (AI) based solutions in public healthcare
settings in India.
We‘d like to thank the Government of Uttar Pradesh and all the functionaries of the District of Bahraich,
who made it convenient to conduct the study at Nanpara, Kaiserganj, Amwahussainpur, Gayghat, Kundsar
and Bhakhla. The support provided by the local administration, despite straining of resources due to COVID
pandemic, is deeply appreciated.
The expanse of this multi-phase study covering both maternal and pediatric illnesses in the context of
community and first level care required a multi-disciplinary team. We are grateful to the entire Research
Team including the Expert Reference Group, the Advisory Group, the Co- Principal Investigators (PIs) and
Research Officers, who took time out from their busy schedules to guide every step of the process and give
valuable clinical guidance for adaptation of guidelines, implementation, and evaluation. We also acknowledge
the inputs received from the multidisciplinary steering group of NITI Ayaog on their valuable inputs on the
possible improvements and technology use in the way forward studies.
We express our deep adulation for the Elsevier team based out of India and UK, whose tireless efforts in
getting the technology up and running, modulating it based on our feedback, and supporting the
implementation of the study, were critical to the success of this project.
This study would not have been possible without the dedicated effort of the members of Piramal Swasthya
Management Research Institute who were involved in the community part of the study. They helped in initial
training of the participants and then hand-held these primary health care functionaries through the study.
They led from the front at the study site, despite the risks of the pandemic, allowing the researchers to
manage this study virtually, which was forced upon us due to pandemic related travel restrictions.
Finally, and most importantly, we express our sincere appreciation for all the Accredited Social Health
Activists (ASHA), Auxiliary Nurse Midwives (ANM), Staff Nurses and Medical Officers from Bahraich
district who participated in the study. They have put in extraordinary efforts during very difficult times and
shown exemplary levels of commitment to adopt an advanced technology like Clinical Decision Support
System (CDSS) to improve the health of their beneficiaries.
We are also thankful to Dr Rajiv Garg and Dr N N Mathur, incumbent Director(s) of Lady Hardinge Medical
College during the course of this study for their approval and support for conducting this important study.
We sincerely hope that the outcomes of this study shall contribute to the development and use of knowledge-
based technologies like CDSS & AI in India. We believe that this intervention can act as a steppingstone to
the wide-scale adoption of Guidelines and evidence-based practices as the country marches to achieve the
Sustainable Development Goals and an overall improvement in the health of every Indian.
Principal Investigators, HAIC Study
Lady Hardinge Medical College and All India Institute of
Medical Sciences, New Delhi
287887/2021/DM&A
489 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
9
July2021
1. Key Stakeholders and Project Team
Group Role Nodal person
Steering Group
(NITI Aayog
Expert
Committee)
Chairman Dr. V.K. Paul, Member (Health), NITI Aayog
Member Dr. Rakesh Lodha, Professor, Department of Pediatrics, AIIMS
Member Mr. Anurag Aggarwal, Director, IGIB – CSIR
Member Dr. Debdoot Sheet, Asst Professor, IIT Kharagpur
Member Dr. P Santanu Chaudhury, Director, IIT Jodhpur
Member Dr. Balram Ravindran, Professor, IIT Madras
Member Prof. Mausam, IIT Delhi
Member Dr. Sudeep Gupta, Director, ACTREC
Member Ms. Anna Roy, Sr Adviser, Frontier Tech Vertical, NITI Aayog
Investigators
and
Expert Review
Group (ERG)
Principal Investigator Dr. Manju Puri, Director Professor& HOD, OBGYN, LHMC
Co-Principal Investigator
Dr. Virendra Kumar, Director Professor& HOD, Paediatrics,
LHMC
Co-Principal Investigator Dr. Varinder Singh, Director Professor, Paediatrics, LHMC
Co-Principal Investigator Dr. Praveen Kumar, Director Professor, Paediatrics, LHMC
Co-Principal Investigator Dr. RM Pandey, Professor and HOD, Biostatistics, AIIMS
Co-Principal Investigator Dr. K. Aparna Sharma, Addl. Professor, OBGYN, AIIMS
Co-Principal Investigator Dr. Shilpi Nain, Professor, OBGYN, LHMC
ERG member Dr. Shubha Sagar. Trivedi, ex-HOD, OBGYN, LHMC
ERG member
Dr. Monika Rana, Director, Directorate of Family Welfare, Govt
of NCT Delhi
ERG member Dr. Harish Pemde, Director Professor, Paediatrics, LHMC
ERG member Dr. Juhi Bharti, Assistant Professor, OBGYN, AIIMS
ERG member Dr.VidushiKulshreshtha, AssociateProfessor, OBGYN, AIIMS
287887/2021/DM&A
490 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
10
Group Role Nodal person
Project Sponsor
NITI Aayog
Ms. Anna Roy, Sr Adviser, Frontier Tech Vertical, NITI Aayog
Sabyasachi Upadhyay, Associate/Research Officer, NITI Aayog
UK-Department of
International Trade
Ed Rose, Edna D‘Souza, Dr. Eisha Anand
UK-Department of
International Trade –
HealthcareUK
Akoto Agyeman, Rasik Tailor, Natalie Bain, Nicola Parry,
May Noonan, Christopher Jordan, Madhukar Bose, Jane Grady
Technology
partner –
Elsevier
UK representative Dr. Robert Dunlop, Clinical Director
India representative
Dr. Ujjwal Rao, Senior Clinical Specialist
Hema Jagota, Director
Project oversight Meera Saini Gupta, Senior Product Sales Manager
Project
Execution
Partner -
Piramal
Swasthya
Management
Research
Institute
Technical Integration
Mr. Devesh Varma, CTO,
NV Lakshmi V, Sr. Manager IT
Field Implementation
Dr. Bhawna Bakshi, State Transformation Manager
Mr. Balmukund Sharma, District Transformation Manager,
Bahraich, UP
Clinical Oversight Dr. Vibhor Kumar, Chief Manager, Quality and Excellence
Project Implementation Research Officers Dr. Jyoti Mishra, Dr. BindiyaPahuja
Writing Group Dr. Manju Puri, Dr. Varinder Singh, Dr. Ujjwal Rao, Meera Gupta
287887/2021/DM&A
491 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
11
2. Introduction
2.1. Background
Maternal and child health are important indicators for evaluating the country‘s effort to provide equitable
access to health and socioeconomic growth. Global trends indicate that maternal mortality rate dropped by
44% worldwide between 1990 to 2015, however, much remains to be done to achieve SDG target of reducing
the maternal mortality ratio below 70 per 1,00,000 live births by 2030
1. As per WHO, it is estimated that
approximately 830 maternal deaths occur per day due to causes related to pregnancy and child-birth related
complications. Out of the total deaths, 99% occur in developing countries, especially in low resource settings
such as rural areas, poorer communities and hard to reach areas
2. Though India has managed to bring down
the MMR from 556 per 1,00,000 live birth in 1990 to 113 per 1,00,000 live births in 2016-2018, it remains one
of major countries with high MMR burden
3. In 2015, India accounted for 15% of all the maternal deaths
4.
Similarly, despite having made significant progress in reducing under-five mortality rate, it is still unacceptably
high in India. As a signatory to meet Development Goals set out by the UN, India is not only committed to
bring down the under-five mortality rate to 25 deaths per 1,000 live births by 2030; but intends to better it
with a declared aim of reaching U5MR of 23 by 2025. Perinatal deaths, Pneumonia, Diarrhoea are most
common causes of deaths to which undernutrition is an important contributor. While the country is making
some headway but there is a wide variation in performance of the various constituent states. As per an
estimate, only 73% of under-five children suspected to be suffering from pneumonia were taken to health
care providers while only 20% of children suffering from diarrhoea received Oral Rehydration Solutions.
Despite significant improvement in institutional deliveries (79%), early initiation of breastfeeding is seen in
only 41%. Wasting, stunting, anaemia is unacceptably high in Indian children
5.
Antenatal Care (ANC) forms one of four pillars of Safe Motherhood Programme which aims to reduce
maternal mortality. The commonly known causes of maternal mortality such as haemorrhage, sepsis,
hypertension, prolonged labour, unsafe abortion, anaemia, etc. are easily preventable through quality of care
during and after birth. Evidence suggest that Antenatal Care (ANC) reduces pregnancy and childbirth related
morbidity and mortality by almost 80% and should begin from the early stages of pregnancy
6. WHO
recommends a minimum of eight ANC visits, ideally at 12, 20, 26, 30, 34, 36, 38 and 40 weeks,with health
promotion including nutrition counselling as one of its important components
7. Regular ANC enables early
detection of high-risk cases, better management of low weight and micronutrient deficient cases, increased
awareness among women regarding pregnancy care and birth preparedness. Indian guideline recommends at
least four ANC visits with first visit preferably in the first trimester
8.
To increase the coverage and access of public health services, the government has started various
programmes focusing on maternal and child health, family planning, and immunization. Few of the significant
initiatives being, introduction of ASHAs in the health systems to act as a link between community and health
facilities, launch of initiatives such as Janani Suraksha Yojana (JSY), a safe motherhood intervention promotes
institutional deliveries to reduce maternal and infant mortalities by providing financial assistance to pregnant
women (below poverty line) agreeing to deliver in government or accredited private health facilities. Janani
Shishu Suraksha Karyakram (JSSK) is an initiative for providing free and zero expense delivery to all pregnant
women delivering in public health institutions. It entitles free transport from home to health facilities, free
drugs and consumables, free diagnostic, free blood, free diet for the duration of a woman‘s stay in the facility
and covers new-born related illness till 30 days after birth. In case of referral, free transport and treatment is
also provided for two ways
9,10.
However, the mere presence of ANC services and initiatives does not guarantee its utilization nor brings
about a positive outcome. Various reasons exist for the gap in ANC coverage such as availability and
accessibility of health services and care providers, socio economic status, education, community awareness
11.
As per NFHS 2015-16, only 58.6% of the mothers in India had ANC check-up in their first trimester whereas
mothers who had at least four ANC visits accounted for 51.2%. The percentage of women receiving full
ANC check (at least four antenatal visits, at least one tetanus toxoid (TT) injection and iron folic acid tablets
or syrup taken for 100 or more days) in India is very low at 21%. Anaemia during pregnancy is associated
with increased risk of maternal mortality. Fifty three percent of women in the age group of 15- 49 years are
287887/2021/DM&A
492 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
12
anaemic in India
12. It is pertinent to note that the quality of primary healthcare is severely affected by the
strain on Primary Healthcare Centres (PHC) where the PHC, usually led by one Doctor, is expected to
provide comprehensive primary care for up to 30,000 residents
13. A major need for optimal care provision is
to improve the availability of skilled human resources in health systems. There is a shortage of 10,907 ANMs
as against the required number of 1,84,160 and there is a shortage of 3,773 doctors at PHC as against the
required number of 25,743. A similar shortfall is experienced in the count of other specialists and health
personals. Other challenges in providing quality of care is poor infrastructure, According to Rural Health
Statistics 2018, as of 31st March 2018, India has a shortfall of SC, PHC and CHC by 18%, 22% and 30%
respectively
14. 66% of the Indian population lives in rural areas
15 and is largely dependent on the public sector
for their medical needs. Process assessment for the health service components has shown gaps and needs to
improve service delivery process, especially adherence to service guidelines by providers
16,17. Lack of timely,
quality, and convenient access to healthcare in the vicinity, compels them to travel long distances to seek care
thus adding to out-of-pocket expenditure.
To address all the factors affecting poor coverage of maternal and child health, efforts ought to be directed
towards leveraging existing capabilities to provide better quality of care and services to improve key priority
indicators. One promising strategy involves ‗task shifting,‘ where front-line, non-physician health workers are
delegated some of the tasks traditionally performed by physicians
18. In the setting of HIV/AIDS care, task-
shifting has been shown to improve health outcomes and processes of care
19. In India, there is some evidence
that task shifting CVD risk assessment to non-physician health workers via a simple algorithm can increase
the detection of CVD
20.
Empowering front-line health workers, and potentially the mid-level healthcare providers, with knowledge-
based technology solutions, such as Clinical Decision Support System, based on Indian Guidelines can serve
as a job aid and provide manifold benefits. It will allow the care provider administer care based on a standard
treatment protocol while ensuring an appropriate triaging with seamless continuum of care; reduction of
unnecessary intervention and inappropriate referrals; and, provision of pre-referral or definitive treatment at
point of care.
There was an established need for localized clinical decision support, which has also been previously field-
tested in Cardio-vascular diseases
18. However, the need for transforming Standard Treatment Guidelines
(STG) into active clinical decision support, including local dialects, to ensure adherence to STGs in maternal
and child health, was yet to be fulfilled.
2.2. Arezzo®
Arezzo® is an active clinical decision support (CDS) and pathway technology that empowers personalised
care delivery at patient and population level. The technology supports the authoring and execution of Level 3
"active" clinical guidelines that are made computer interpretable through Declarative Artificial Intelligence
(AI). By integrating with clinical care systems, Arezzo® matches evidence-based guidelines with patient and
disease information and dynamically evaluates best-practice treatment options specific to the patient at that
stage in their care. Arezzo® is owned by Elsevier.
Arezzo® CDS proprietary software consists of the following components:
Arezzo® Composer – for configuration of CIGs content
Arezzo® Bridge – for configuration of display-logic, including patient summaries, referral letters,
and to facilitate data-entry
Arezzo® Performer – the high-performance AI engine that processes CIGs content against patient
data to deliver ongoing recommendations, dynamic order sets, and alerts.
Arezzo® Conductor – middleware component that manages external interfaces. In the pilot project,
Arezzo® Conductor will provide the ReST API that is consumed by the mobile app.
MS SQL database for storing the Arezzo® CDS state files and audit data.
In this study, the Arezzo® CIGs were delivered through an Android-based app that was installed on mobile
tablet devices. The mobile app communicated online with the Arezzo® ReST API and dynamically
287887/2021/DM&A
493 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
13
renderedthe outputs to healthcare workers. Outputs included requests for more data, decisions, and tasks.
The app then sent user inputs back to Arezzo® for processing.
Arezzo® was validated through several studies in the UK, Europe, and New Zealand. The details of these
validation studies can be found in Annexure 1.
2.3. Rationale
Arezzo® canenable transformation of primary healthcare by implementing multiple, merged STGs at the
point-of-care, where the FHWs will only see content relevant to the patient. A longitudinal clinical pathway,
based on the State Agent of Arezzo®, will be generated for every unique beneficiary, starting from her
doorstep. Arezzo® will also record the decisions taken for monitoring compliance and variance from STGs,
to automatically generate data about adherence to STGs.
The platform was envisaged to bridge some of the key gaps and challenges in effective healthcare delivery in
the public healthcare delivery system of India. Some of these include:
a. Inadequate and inappropriate referrals from health outreach
b. Knowledge gap in FHWs
c. Non-availability of local language guidance for FHWs
d. Lack of evidence-based primary health screening
e. Inaccurate operational and clinical data for evaluation of compliance measures
f. Lack of health outreach information in the care pathway, which will be addressed by generating a
unique pathway for a beneficiary which could be integrated with the Electronic Health Record
(EHR) in the future
Arezzo® generates high-quality clinical data sets during usage. Whilst such data lends itself to post-hoc
analytics and machine-learning, algorithms can be used to generate potential new clinical insights, but these
must be validated before being incorporated back into CIGs.
It was expected that a secondary gain from a wide-scale Arezzo® implementation in India will be the
significant increase in district-level data about care delivery, referrals, and outcomes, including the potential to
partially automate NFHS-like national surveys.
3. Study Aim and Objectives
The purpose of the study is to test the feasibility of Elsevier‘s ―Arezzo®‖, with adapted Indian Standard
Treatment Guidelines, used in public health care settings through a Pilot intervention.
The study will focus on the following objectives:
a. To test the validity and accuracy of Arezzo® content after its customization based on Indian STGs
demarcated for various levels of care in maternal and child health.
b. To study the feasibility of ―Arezzo®‖ in primary and secondary healthcare settings.
c. To evaluate the following with the addition of Arezzo® in public healthcare settings:
Usability
Usefulness
Adherence to STGs
287887/2021/DM&A
494 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
14
4. Hypotheses
a. Integration of ―Arezzo®‖ - an active Declarative AI clinical decision support system is usable and
useful to health workers in primary and secondary healthcare settings.
b. Provision of Arezzo® customized for Indian guidelines-based recommendations facilitates
adherence to primary health screening guidelines.
5. Methodology
5.1. Study Phases
The study was carried out in two broad phases. Phase 1 was to digitise and transform the STGs to CIGs for
incorporation into the CDS tool and then to assess the validity (Phase 1a) and the reliability (Phase 1b) of the
of the converted guidelines (from STGs to CIGs) in the CDS tool. Phase 2 was to test feasibility of the tool
in terms of the usability, usefulness, and adherence to STGs with its usein the field for Primary Healthcare
and community healthcare workers (CHC, PHC and sub-centre level).
These phases were conducted as per the following timelines:
Preparatory phase: Content conversion and incorporation into CDS (Nov 19-Feb 20)
Phase 1 a: ContentValidity - March 2020
Phase 1b: Content Reliability - April through September 2020
Training for Phase 2 - 28-30 September 2020
Phase 2: Implementation at Study Site (Bahraich, UP) - October 2020 through March 2021
Study data collection - 21-Oct-2020 to 8-April-2021.
Report Writing
Final report submission to sponsors May 2021
Presentation to NITI multidisciplinary steering group and discussions for inputs
HAIC Pilot Project Report (updated post inputs from the steering group)
Figure 1: HAIC Study GanttChart
The phase I was conducted at the department of Obstetrics & Gynaecology and Paediatrics at Lady Hardinge
Medical College while the phase II was conducted in the community settings (Bahraich district in the state of
Uttar Pradesh). The selected district is one of the aspirational districts of NITI and had the presence of a
health support agency, who were taken on board for the study as implementation partner.
Project Milestone31017241815222961320273101724291623306132027411182518152229613202731017243171428512192629233071421284111825181522181522181522181522181522
Arezzo-AMRIT Integration
Ethics Approval
Pediatric Content
Maternal Content
Hindi translation ASHA/ANM
Content Validity (Phase 1a)
Content Reliability (Phase 1b)
Phase 2State ApprovalBaseline SurveysPhase 2Report writing
NITI Aayog update meetings
Final NITI steering group review
MarOctMay Jun
201920202021
AprMarNov Dec Jan FebApr May Jun Jul Aug Sep
1-Jun-21
Nov Dec Jan
26-May-2012-Nov-20
Feb
Arezzo -AMRIT Integration; Installation at NIC
Pediatric Content Conversion
Maternal Content Conversion
Phase 1b
LHMC Ethics ApprovalAIIMS Ethics App
Phase 1a
Translations to Hindi
287887/2021/DM&A
495 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
15
5.2. Target Health Workers
Computer interpretable guidelines (CIGs) using the Arezzo® platform were demarcated, and three levels of
decision-making were programmed for the three rolesviz:Accredited Social Health Activists (ASHA) workers,
Auxiliary Nurse Midwives (ANMs) and Medical Officers (MOs) (Table 1).
Table 1: Target Health Workers
Health Worker Role
Accredited Social Health Activist (ASHA) Front-line health screening
Auxiliary Nurse Midwife (ANM)/Staff
Nurse
Primary care screening and interventions as indicated
Medical Officer (MO) Primary and secondary care
5.3. Constitution of Expert Reference Group
An ExpertReview Group (ERG) of Obstetricians and Pediatricians was constituted for this study (Section 2 –
Key Stakeholders) to provide their expert opinion throughout the duration of the study. The Principal
Investigators (PIs) and co-PIs led the ERG and provided directions on the design and execution of the study
as well as the analysis of the study results. Two Research Officers were also onboarded to help conduct the
study. Elsevier incorporated the STGs into the Arezzo application such that they were demarcated for
various levels of health service providers i.e.MOs, ANMs and ASHAs with the support of the research team.
5.4. Preparatory Field Visit
In December 2019, team members visited Bahraich, UP to understand the workflow at the primary
healthcare facilities and to gain insights for content conversion and development of the solution.The team
met ASHA, ANMs and MOsat 2 pre-selected Primary Health Centers (PHCs), 1 sub-center, 2 Village Health,
Sanitation and Nutrition Day (VHND) clinics and 1 Community Health Center (CHC). Our implementation
partner, Piramal Swasthya Medical Research Institute (PSMRI) helped facilitate the visit including the
meetings. The salient insights were that CHC had high volumes and was the preferred place to go by the
villagers. The infrastructure of the PHCs was impressive but PHC and the sub-center experienced low
volumes of patients. The PIs were impressed by the knowledge and capability of the ASHA workers and the
ANMs. In the maternal population, incidence of anemia was high, but interestingly there were very few cases
of gestational diabetes. This reflected their practice as they did not routinely do screening for GDM.
5.5. Infrastructure and Resources
Android-based tablets with 4G internet connectivity and a pre-installed instance of the AMRIT (Accessible
Medical Records via Integrated Technologies) android-based mobile app, developed by the implementation
partner PSMRI (Piramal Swasthya Management Research Institute), that was integrated with Arezzo®. This
device was used by the Research Officers and the selected health workers enrolled in the study.
5.6. Selection and transformation of Guidelines
As a pilot project to test the hypotheses 10 conditions, 5 each from maternal and child health (Table 2) were
initially finalized through a consensus of the Expert Reference Group (ERG). These guidelines were then
digitized and transformed for use with Arezzo CIGs.
287887/2021/DM&A
496 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
16
Table 2: Guidelines selected for the HAIC Pilot
Condition Guideline
Maternal Health
1.
Antenatal care (ANC) for
uncomplicated pregnancy
Guidelines for ANC Skilled Attendance at Birth by ANMs LHVs
SNs 2010 MOH GOI
ASHA Skills That Save Lives – Module 6 and 7
2.
Gestational diabetes
mellitus (GDM)
Diagnosis and Management of Gestational Diabetes Mellitus -
Technical and Operational Guidelines, NHM, MOH, GOI, 2018
3.
Hypertension in
pregnancy including
eclampsia
Guidelines for ANC Skilled Attendance at Birth by ANMs LHVs
SNs 2010 MOH GOI
4. Anemia in pregnancy
Anemia Mukt Bharat - Intensified National Iron Plus initiative (I-
NIPI) Operational Guidelines, MOH, GOI, 2018
5. Postnatal Care ASHA Skills That Save Lives – Module 6 and 7
Child Health
6. Feeding
National Guidelines on Lactation Management Centres in Public
Health Facilities
Guidelines for enhancing optimal and young child feeding
practices, MOHFW, GOI, 2013
ASHA Skills That Save Lives – Module 6 and 7
IMNCI package for young infant and child
7. Immunization
National Immunization Schedule (NIS) for Infants, Children and
Pregnant Women, MOHFW, GOI, 2017
8. Pneumonia
IMNCI package for young infant and child
Facility Based Care for Sick Children, NIPI, LHMC
9. Diarrhoea
IMNCI package for young infant and child
Facility Based Care for Sick Children, NIPI, LHMC
10. Care of the sick neonate
Facility Based New-born Care Operational Guideline, MOHFW,
GOI, 2011
Home-Based New-born Care Operational Guideline, MOHFW,
GOI, 2014
IMNCI package for young infant
5.7. Detailed Methodology
5.7.1. Phase I
The validity and reliability of the content was tested at the Obstetrics andGynaecology and Paediatric
departments of Srimati Sucheta Kriplani Hospital and Kalawati Saran Childrens‘ Hospital associated with
Lady Hardinge Medical College, andsupported by co-PIs atthe All India Institute of Medical Sciences, New
Delhi.
The purpose of this phase was to assess the validity and the reliability of the of the converted guidelines
(from STGs to CIGs) in the CDS tool. This was done in two parts:
287887/2021/DM&A
497 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
17
Phase 1a: Content Validity
The PIs tested the various steps and components of the individual standard treatment guidelines
(STGs) converted into computer interpretable format (CIGs). both for content (through static
interface) and logical flow (through web-based tester). After each testing session, feedback was sent
for fixing of errors or improvisation to the Elsevier content transformation team and the content was
re-tested again after amendment by Elsevier. Separate pathways were created for ASHA, ANM and
MO keeping in mind their training and expected skill sets. These were further fine-tuned to offer
actions keeping in mind the place of contact (CHC vs. PHC vs. SC vs VHND vs home visits). At all
times, the fidelity to the guidelines was ensured at the concerned levels of care.
The transformation of paper based STGs into computer CIGs went beyond simple digitization. It
included harmonization of overlapping symptoms from the different guidelines and creating AI for
stepwise optimization and unfolding of the action-oriented outcomes from the entered patient data.
Flow of the guidelines had to be adapted according to the flow of real time conversation and
interaction of the HCW with the patients to elicit relevant history related to various
conditions.Dummy patient data was used to test all segments of the CIGs separately for each of the
target health worker roles.
Once all the subcomponents were tested, the whole program was retested using the entire content
together, for example with complex patient journeys that involved multiple symptoms or conditions
across several STGs. Feedback from the retesting was then used to rearrange the CIGs workflow as
required.
Phase 1b: Content Reliability
Ethics committee approval was obtained prior to the commencement of phase 1b.
Phase1b was also conducted in the controlled tertiary care setting at LHMC, using prospective data.
For this phase, the Research Officers weretrained to use Arezzo® tool, now incorporating the CIG
based on National STGs, through a training workshop. They were given mobile tablets equipped
with Arezzo® tester and internet connection for the trial period.
The system was tested prospectively by Research Officers (equivalent to the rank of MOs and
Nurses) under the guidance of the Expert Review Group clinicians using an inter-rater agreement
between the CDSS based decisions (as arrived by the Research Officers) and clinical assessment and
management by ERG clinician.
After each testing session, feedback was sent for fixing of any remaining errors or for improvisation
to the Elsevier content transformation team and the content was re-tested again after amendment by
Elsevier.
Local language conversion in Hindi for standard treatment guideline content relevant to front-line
health workers (ASHA/ANM) was done in parallel to testing in this phase. The testing was done in
English and then in Hindi for the ASHA and ANM/SN content.
Milestones and course adjustments during Phase I:
The salient milestones of this exercise comprised of:
Identification and correction of errors in the CIG content when the test flow did not match the
source STG content.
Adaptation of CIGs to the context of Indian patients; integrating the various vertical guidelines to
cover co-morbidities and to cover different age groups (infant and young child)
Clarifications of STG content: in certain places the flow on the paper guidelines was not clear
andneeded to be articulated/clarified further.
Requirements for new content that were not contained in the source STG but were required to
ensure the appropriate clinical workflow were fulfilled.For example, the obstetric expert team
provided list of questions for:
o taking the obstetric history by a medical officer,
o estimation of duration of pregnancy when the woman cannot recall her LMP
287887/2021/DM&A
498 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
18
Requirements or reorientation for new content that was considered important to cover service areas
and skill set of the peripheral functionaries but was not covered in the selected STGs.
o The guideline cover on Sick New-born was reoriented to cover the Integrated Management
of New-born and Childhood Illness (IMNCI) based care of the sick young infant instead of
facility-based guidelines keeping in mind the job profile, training and skill sets of the target
health functionaries. In addition, home-based New-born care package – a critical
community New-born care intervention- was also included (tested in the healthy New-born
babies in the postnatal ward).
o Emergency Triage and Treatment (ETAT) component of the Care of the Sick child to cover
for cases other than Pneumonia and Diarrhoea for appropriate assessment and emergency
management and/ or referral was created for MOs.
o Developmental assessment for children was included to completely harmonize with the
Mother and Child Protection (MCP) Card.
Harmonization and updating with new guidelines that have been published since the STGs were
created
o For example, the recently published paediatric guideline on assessment and management of
respiratory disease
o Guidance on newer contraceptives: DMPA, Centchroman and PPIUD
Expert opinion on managing the relationships between multiple CIGs in the same patient for
prioritization of treatment
o For example, hypertension in a pregnant woman is more important to manage urgently
compared with anaemia that is not severe.
Information about service provision within the target health district (Bahraich)
o For example, gestational diabetes screening with oral glucose loading test is not performed
routinely
o Also, information about role-based activities enabled the STG content to be separated by
role in the CIGs
In addition, the study plan was impacted severely by the reorganisation of the hospital services due to
COVID. For a significant duration of time, the OPDs had been closed and even when they opened in a
phased manner, the interactions were affected due to the users not willing to stay in the hospital for any
longer than their primary needs for the visit. Travel related restrictions during lockdown, issues related to
worker safety and similar challenges made us make amendments to study plan and the phase 1b was
conducted using the case records of the patients under treatment instead of shadowing of real life OPD
interaction as was originally planned. Many more interactions than the originally planned were done during
this forced extension to extensively validate the digitised care pathways.
Overall, the Principal Investigators of both, the maternal and paediatric teams, met several times face to face
and virtually to review the content. In addition, sub-component testing was also done individually by the PIs
and research officers. The total number of hours spent collectively by the PIs for content conversion and
phase 1 review was over 500 hours. The testing case mix for Phase 1 is detailed in Annexure 4.
Phase 1 study report was submitted and presented to the Expert Committee of NITI Aayog, whose express
approval along with the Ethics Committee approval met the pre-requisites for kicking off Phase 2.
5.7.2. Phase II
This phase involved testing of the feasibility of implementing Arezzo® in primary and secondary healthcare
settingsin the district of Bahraich in Uttar Pradesh, India. The study-site selection was based on discussions
and consensus with NITI Aayog and PSMRI, the project implementation partner.Two CHCs of Kaiserganj
and Nanpara and their associated PHCs were selected,based on feasibility, for the study.
287887/2021/DM&A
499 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
19
Study Participants
50 health functionaries were approached to request voluntary participation in the study. One MO did not
consent to participate in the study. Remaining 49,whoparticipated in the study belonged to the following
cadres: Medical Officers: 8;Staff Nurses:7;ANMs: 14: and ASHAs: 20. Over the course of the study three
ASHAs withdrew from the study for personal reasons and were replaced by 3 other ASHAs from the area.
These changes occurred during the months of November and December‘ 20. From January‘ 21 onwards
there was no further change in the participants. Of the eight MOs enrolled, three did not actively participatein
doing the cases in the digital mode and evaluating the CDSS application. As a result, they also did not
provide their feedback in the final scorecard. A baseline survey was conducted to document the work profile,
language and smart mobile use proficiency and training of the participants.
Table 3: HAIC Study Participants
CHC Nanpara Kaiserganj
Medical Officers 2 3
Staff Nurse 2 2
PHC Amwahussainpur Gayghat Kundasar Bhakhla
Medical Officers 1 1 0 1
Staff Nurse 1 2 0 0
ANMs 4 3 3 4
ASHAs 5 5 5 5
Training
Astravel to the field was severely impacted due to the COVID-19 lockdown, training was conducted virtually
by the PIs, the research officers and Elsevier team. The field staff of PSMRI provided the handholding and
training of the staff on the ground. Training was done over three days from 28-30 September 2020. Initially,
the training was conducted only for ASHA and ANMs. Medical Officers were introduced to the tool later in
November 2020.
Training Modules:
Study overview
Participant information sheet.
Informed consent
Mobile device tablets distribution.
Hands-on training of AMRIT application and CDSS
After the training, the workers were asked to have familiarisation and run-in with the use of application using
test data for 2-3 weeks. The workers were regularlycontacted and encouraged as well as facilitated to use the
app. The study was officially rolled out on 21-October-2020.
Course of the Phase 2 Study
During the initial phase of the phase 2 roll-out the uptake of the application was slow. Internet
connectivity was a major hurdle in many places. Another challenge was related to adoption of the
AMRIT mobile application. The initial plan was to roll out AMRIT one month before Arrezo
clinical decision support system (CDSS) to facilitate familiarization of the HCWs with the app and
the registration of the beneficiaries residing in the area. However, these timelines did not materialize
due to the Covid-related lockdown and AMRIT and Arezzo CDSS were launched at the same time.
As a result, the workers were unfamiliar with usage of the AMRIT, the enumeration application and
were not able to record any interaction with the beneficiary on the digital platform or access CDSS
for the guidance on such an interaction. The internet issues were solved by selecting alternative
287887/2021/DM&A
500 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
20
functional network provider after a one-on-one discussion with the workers. In addition, where the
internet issues persisted, the users were asked to note down the data while in the field and enter it
into the app and complete the CDSS journey later in the day when they reached any area with better
internet connectivity.All these interactions (care seeking journeys) wereanalysed by research officers
and HCWs were regularly contacted by them for easing out any teething troubles and
troubleshootingissues related to technical and/ or content aspect of the programme.
Several issues cropped on integration of Arrezo CDSS with AMRIT, e,g. timing out of the CDSS
server on not receiving timely inputs which were either due to learning curve or slowness of the
connection or due to bugs arising in either of the two programmes. These issues were flagged,
analysed for the cause and resolved in a systematic manner. Both the teams - Elsevier as well as
PSMRI IT team supporting AMRIT - had to work in tandem to sort these.
One month after the roll-out, the research officers and Elsevier team visited the field. This visit
occurred from 22–25 November 2020. During this visit, many training gaps emerged. We found
that the relatively fast-paced virtual trainings had prevented the workers from truly grasping the
concept of the CDSS application and how it was to be used. This visit, therefore, was utilised for
supervised interactions, identification of problem areas and individual feedbacks and reinforcements.
Joint sessions were also held in smaller groups where all the identified problem areas were explained
to the workers.
One of the key outcomes of the app and CDSS tool was to allow identification of high-risk patients,
appropriate referral and step-up care. This required mapping the continuum of care from ASHA to
ANM to MO. However, the functionality to―Get Data‖from one worker to the other inline superior
had problems e.g.ANM was not able to see the beneficiary details of the beneficiary registered by the
ASHA and likewise the MO was not able to view beneficiary details that the ANM or ASHA had
entered. PSMRI technical team worked hard to resolve the integration issues and it required a lot of
fixing, re-fixing and time to get sorted. Each build helped to make the AMRIT application more
stable and the integration more reliable. In all, three successive app builds were released in an
evolutionary Agile approach. Each time a new built was released all the tablets had to be physically
collected from each worker from the field and brought to head office of Piramal at Lucknow, to be
updated and then redistributed. This was a time-consuming exercise.
A final buildincorporating all the desired enhancements was released beginning February2021.
Starting from 7th February 2021, the journeys completed by the workers were analysed almoston
adaily basisand feedback communication with the workers was continued.
Anonymous online surveyswere launched to get feedback from all the users with regards to the
usability and usefulness of the application twice in phase II (one during the 8-18 March ‗21 and
another final one during the first week of April 21. These two surveys had identical questions but
were repeated as the assessment of the usefulness and usability in first survey could have been
affected as the app was not fully stabilized in the initial weeks and many workers who were slow to
adopt the technology had not used the app often enough. We hypothesized that the second survey
which was nearly 8 weeks after the release of the stabilized version of the app wouldallow to get a
better more realistic assessment of its usefulness and usability. The trend of app usage and the access
to STG supported this hypothesis as most of the incomplete journeys (recorded patient interactions)
were mainly seen at the start of the study due to user hesitation, learning curve and technical reasons
like poor or no internet connection, server timing out due to slow inputs/internet, software bugs, etc.
As the questionnaire design of the score card did not allow any detailed understanding of the users‘
view, we invited all of them to participate in smalllocation-based focus group discussions (FGD). It
helped to get detailed insights into their feedback on theusability and usefulness of the solution, its
limitations, and possibilities for future benefits.
46 out of 49 participants responded to the survey. 3 medical officers were not able to respond and
provide their feedback. The following section only details the responses from final survey while the
findings of the midterm survey are detailed in annexure 5. The focus group discussions were
executed from 6-10 April 2021 and the principal investigators were intimately involved in each of the
six focused group discussion sessions that have been detailed in Section 5.1.3.3.
287887/2021/DM&A
501 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
21
The final report hasused parsed data,as compared to the interim report submitted in March 21,
excluding duplicate and/ or unlinked entries forced due to anonymization required for the CDSS
tool, in view of data privacy rules.
The overall process is described in the following flowchart (Fig 2):
Figure 2: HAIC Study Flow-Chart
Sampling
Based on a study and field review by the Principal Biostatistical Investigator, a convenience sampling
methodology with a sample size of 3000 journeys was finalized. Because this was a field study, there was no
requirement for collecting a certain number of case journeys per guideline. Workers were asked to register
and run Clinical Decision Support on a minimum of 5-8 cases a week.
Inclusion criteria
Pregnant women seeking antenatal care or postnatal care and Children under 5 years, with any of the
conditions included in the pilot (section 4.6) in contact with a health worker in community setting or
those attending a PHC/CHC
5.8. Outcome Measures
The following outcome measures, based on operational criteria specific to the project, as well as clinical
evaluation criteria for maternal and childhealth were evaluated.
287887/2021/DM&A
502 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
22
Table 4: HAIC Study Outcome Measures
Phase
Outcome
Measure
Type of
Data
Evaluation Criteria Data security
Phase 1a Content validity Qualitative Concordance by PIs Non-identifiable data
Phase 1b
Content
reliability
Qualitative Concordance by PIs
Anonymized patient data
Secure transmission of data
through encryption
Phase 2
Usability
Quantitative
Qualitative
Anonymized
Quantitative score
card surveys
@
FGD
Written informed consent
Usefulness
Quantitative
Qualitative
Anonymized
Quantitative score
card surveys
@
FGD
Written informed consent
STG Adherence Quantitative
Cumulative STG
Adherence Trend
STG Adherence
indicators
Pseudonymized patient data
Secure transmission of data
through encryption
SSL certification
DNS over HTTPS
@Score card-Annexure2
5.8.1. STG Adherence Indicators
The following STG adherence indicators were measured through automated data collection via Arezzo:
Table 5: STG Adherence Indicators
Condition STG Adherence Indicators
Maternal Health
Antenatal care (ANC) for
uncomplicated
pregnancy
1. Documented LMP
2. Documented EDD
3. History taken (specific symptoms, systemic illness, drug intake,
family history of systemic illness etc.)
4. Weight measured
5. Blood Pressure taken
6. Blood sample taken (for Hb)
7. Blood sample taken (for blood group, Rh)
8. Blood sample taken (for VDRL)
9. Urine sample taken (for protein)
10. Urine sample taken (for glucose)
11. Abdomen examined
12. IFA supplements prescribed
13. Tetanus Toxoid prescribed
14. Calcium supplements prescribed
15. Anthelmintics prescribed
16. Counselling done (dietary advice)
17. Counselling done (contraception)
18. Counselling done (violence)
19. Counselling done (rest)
20. Documented referral when indicated
Gestational diabetes
mellitus (GDM)
21. Screening for GDM done
22. History taken for signs & symptoms of hyperglycemia.
23. OGTT done for screen-positive women
287887/2021/DM&A
503 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
23
Condition STG Adherence Indicators
Hypertension in
pregnancy including
eclampsia
24. Documented history (hypertension or pre-eclampsia/eclampsia in
the previous pregnancy or current pregnancy or family history)
25. Two consecutive BP readings taken four hours or more apart if
SBP ≥ 140 mmHg or DBP ≥90 mmHg or more
26. Test for urine albumin in case of high BP
Anemia in pregnancy
27. Documented history suggestive of anemia (pallor, palpitation,
fatigue)
28. Hemoglobin test at each visit
29. IFA supplements – one tablet twice a day
Postnatal care
30. Iron prescribed
31. Contraceptive advice given
32. Nutrition advice given
33. Breast feeding advice given
Child Health
Feeding
34. Breast feeding within one hour of birth
35. Counselling done on lactation support, breast feeding, IYCF and
KMC (as part of LSU requirements), on complementary feeding
and done on feeding and recommendations during sickness and
health
Immunization
36. Documented immunization history as per National Immunization
Schedule (NIS)
37. Recommendations for vaccination as per NIS
Pneumonia
38. Documented assessment (signs of pneumonia and pneumonia
severity)
39. Documented referral when indicated
Diarrhoea
40. Documented history (feeding, SAM, diarrhoea, blood in stool)
41. Documented assessment (persistent diarrhoea, dysentery, signs of
dehydration)
42. ORS packet prescribed when indicated
43. Zinc prescribed when indicated
44. Documented referral when indicated
45. Antibiotics prescribed
Care of the sick neonate
46. Documented history (preterm, LBW, intrapartum/post-partum
complications)
47. Documented assessment (ETAT)
48. Documented assessment (feeding, crying, breathing, icterus,
pallor, cyanosis, abdominal distension, neck rigidity)
49. Documented referral when indicated
General
50. Documented birth weight
51. Weight measurement on every visit
52. Temperature measurement
287887/2021/DM&A
504 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
24
6. Results
6.1.1. Base-line Survey
A baseline survey was conducted to understand the current practices of these field workers.
While Hindi language proficiency was 100% across all three groups, it was evident that English
proficiency was much higher in ANMs as compared to ASHAs, with 100% proficiency in MOs.
All three cadres of health workers were familiar with STGs and were trained on newly published
guidelines.
The comfort level with mobile devices was highest in ANMs followed by MOs and then ASHAs.
However, none of them used any mobile devices for any routine work.
The perceived benefit of CDSS was highest for ASHAs, followed byMOs, and thenANMs.
Figure 3: Baseline Survey Results
Further details of the baseline survey are contained in Annexure 3.
6.1.2. Phase 1 Results
Phase 1a: Content Validity
All the content related to common illnesses and conditions selected for the pilot study could be
accurately digitized with separate pathways for ASHA, ANM and MO keeping in mind their training,
skills and job profile.
The AI could allow accurate stepwise optimization and unfolding of valid and accurate action-
oriented outcomes from the entered patient details, keeping in context the place of contact (PHC vs.
CHC vs.VHND vs. home visits).
The fidelity to the guidelines was maintained at all times and at all levels of care. It worked equally
well with complex patient journeys that involved multiple symptoms or conditions across several
STGs.
Phase 1b: Content Reliability
Reliability Testing was done on 439 cases (208 maternal and 231 paediatric) as detailed in the Table
below. It coveredmore than the proposed cases that were to be tested for both Maternal and
H I N D I
P R O F I C I E N C Y %
E N G L I S H
P R O F I C I E N C Y %
S T G A W A R E N E S S
%
C O M F O R T L E V E L
W I T H M O B I L E
D E V I C E S %
P E R C E I V E D
B E N E F I T O F
C D S S %
100
15
100
70
30
100
81
100
81
5
100100100
78
11
Baseline Survey
ASHA ANM MO
287887/2021/DM&A
505 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
25
Paediatric guidelines covering the different health functionary roles. Multiple, relevant journeys were
run on these cases, separately for each of the roles (ASHA, ANM and MO). In total, we evaluated
963 maternal and 2172 paediatric journeys. The details of the cases are given in Table 6 below and
the breakdown of journeys is provided in Annexure 4.
Table 6: Phase 1 Reliability Testing Journeys
Each Case completed for all the three roles ASHA, ANM and Medical Officer.
Maternal Cases (N=208)
ANC-without any complications 63
ANC with 1 or more complications
Anemia in Pregnancy
Preeclampsia/Hypertension/Eclampsia
Gestational diabetes mellitus
105
42 *
38 **
33 ***
Post-natal 40
Pediatric Cases (N=231)
Feeding and Immunization alone 33 #
Pneumonia
Pneumonia alone
With feeding and immunization
24 ##
11
13
Diarrhoea
Diarrhea alone
Diarrohea w SYI
With sick child
With pneumonia
With feeding and immunization
76 ###
31
10
6
5
24
Care of the sick child
Care of the sick child alone
With feeding and immunization
44
22
22
Sick Young Infant 32
New-born Care 22
* Includes women with anaemia alone and also women with anaemia and or
preeclampsia/ GDM
** Includes women with preeclampsia alone and also women with preeclampsia and or
anaemia/ GDM
*** Includes women with GDM alone and also women with GDM and /or anaemia/
preeclampsia.
# In total, 92 cases each were done for Feeding and Immunization (rest depicted
elsewhere in the table as and when they were overlapping with other guidelines; the
feeding and immunisation does not get assessed in severe cases as referral takes
precedence over these assessment)
## Includes children with Pneumonia alone and also pneumonia with feeding and
immunization.
### Includes children with diarrhoea alone and diarrhoea for sick young infant, sick
child, pneumonia and diarrhoea with feeding and immunization
287887/2021/DM&A
506 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
26
At the end of phase 1b, all the maternal and paediatric conditions for whom content conversion was
done and AI was created have been tested. The CDSS content on the integrated software (Arezzo-
Amrit app) was tested and retested on a sufficiently large dataset (Table 6) and all the bugs identified
were removed. The software has achieved the capacity to effectively and seamlessly providing
individualized intervention or management solutions with complete reliability, based on the
beneficiaries‘ case details provided.
Furthermore, this fine-tuning of content was coupled with further need based enhancements.
Notably the application is also able to do the following:
o create necessary clinical summaries when beneficiaries are referred from the ASHAs to
ANMs and from front-line health workers to the medical officers.
o covers the early development screening for children up to 3 years.
o supplemented with the addition of images and harmonized with the content of the mother
and child protection card introduced recently in the functioning of front line workers.
The ERG led by PIs reviewed the journeys and agreed that the converted guidelines were sufficiently
and satisfactorily tested for validity and reliability according to the pathways outlined. STG content
was found to be fully transformed and supplemented with more recently published
recommendations, where deemed essential by the experts. The content was further harmonized and
fine-tuned across guidelines using advisory group opinion.
The ERG led by PIs had concordance on the sufficiency of testing to cover majority of the likely
case scenarios for the selected conditions having significant impact on maternal and child health in
the primary setting.
Further, there was concordance on the ability of the configured and optimized version of the
Arezzo® Clinical Decision Support Systems integrated with AMRIT to reliably and accurately
provide individualized treatment / care decision, appropriateto the role of the health functionary and
in complete fidelity with the adopted National guidelines.
6.1.3. Phase 2 Results
In Phase 2 the computer interpretable guidelines and the CDSS were available to use by the health workers
through the app, we used a mixed method approach and collected both quantitative data as well as qualitative
data. The user‘s experience was assessed quantitatively using anonymised survey using scalar scale score cards.
Adherence to the standard guidelines was assessed quantitatively through backend usage data.
Qualitative data was collected through focus group discussions done in smaller batches after the score card
surveys were submitted to delve deep into the reasons for the scoring and get insights into challenges and
desired modifications, if any.
6.1.3.1. Quantitative Scorecard Surveys Analysis
Statistical analysis of the end of study scorecard results was performed using KNIME Analytics Platform
(v4.3.2) and Microsoft Excel for Microsoft 365 MSO and detailed below (Fig 4).
The cumulative positivity rate (rated as agree and strongly agree) for questions related to
Usabilitywas76.09%(Mean Likert Score 3.99; 95% CI: 3.41 – 4.57).
The cumulative positivity rate (rated as agree and strongly agree) for questions related to
Usefulnesswas 80.43%(Mean Likert Score 4.03; 95% CI:2.38– 4.77).
The cumulative positivity rate (rated as agree and strongly agree) for questions related to Overall
Satisfaction was 76.09%(Mean Likert Score 4.10; 95% CI: 3.80 – 4.39).
287887/2021/DM&A
507 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
27
Figure 4: Quantitative Scorecard Survey Results Summary(Phase 2)
The slightly higher cumulative positivity rate for Usefulness indicates the participants inclination towards the
features that they found valuable in their routine tasks, but this wasn‘t necessarily supported by an equal
inclination towards the ease of use of the application, reflecting on its Usability, that wasinitially impacted by
factors relating to technology, network availability and integration with the PSRMI AMRIT app. FGDs
provided further insight into the reasons for these differences and are discussed later.
Usability
The cumulative positivity rate where respondents agreed or strongly agreed on the Usability of CDSS-
integrated mobile was 76.09%(Mean Likert Score 3.99; 95% CI: 3.41– 4.57) with statistically significant
responses to all questions on usability (p <0.05;one-sample t-Test).
The cumulative positivity rate and Mean Likert Scores for each question on Usability were as follows:
The mobile application iseasy to use: 89.13%and 4.33(95% CI: 4.08 - 4.57)
The necessary information and summariesare organized and displayed in a logical manner: 86.96%
and 4.13 (95% CI: 3.91 - 4.35)
I can accomplish my tasks more easily: 73.91% and 3.98 (95% CI: 3.66 – 4.30)
Recording patient data and navigating through the workflow is seamless: 73.91% and 3.91 (95% CI:
3.61– 4.21)
Feel comfortable using it in the social or community setting: 69.57% and 3.85(95% CI: 3.55– 4.15)
It integrates and fits easily into my daily routine: 63.04%and 3.72(95% CI: 3.41 – 4.03)
There was also a statistically significant variance between different groups for the question ―It integrates and
fits easily into my daily routine‖ (p <0.05; One-way ANOVA) with mean Likert Scores for four groups,
showing a higher positivity rate amongst ASHAs compared to the other groups, as follows:
ASHA: 4.2 (95% CI: 3.75 – 4.65)
ANM: 3.21 (95% CI: 2.58 – 3.86)
Medical Officer: 3.20 (95% CI: 2.64 – 3.76)
Staff Nurse: 3.71 (95% CI: 2.83 – 4.59)
There was no statistically significant variance between different groups of ASHAs, ANMs, Staff Nurses and
Medical officers for the other questions on usability.
0%
20%
40%
60%
80%
100%
Usability
Usefulness
Overall Satisfaction
Usability, 76.09%
Usefulness, 80.43%
[CATEGORY
NAME], [VALUE]
USERS WITH
CUMULATIVE
POSITIVE SCORE
RATING
Cumulative Positivity Rate
287887/2021/DM&A
508 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
28
Figure 5: Usability Survey Results
Figure 6: Usability Survey Stacked % Results
5
8
4
4
2
6
9
9
7
4
4
16
17
19
20
26
19
18
12
13
14
14
22
051015202530354045
Icanaccomplishmytasksmoreeasily
Itintegrates/fitseasilyintomydailyroutine
Ifeelcomfortableusingitinsocial/communitysetting
Recordingpatientdataandnavigatingthroughthe
workflowisseamless
Thenecessaryinformationandsummaryisorganizedand
displayedinalogicalmanner
Itiseasytousethismobileapplication
Usability (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
0%
20%
40%
60%
80%
100%
Usability % (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
287887/2021/DM&A
509 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
29
Usefulness
The cumulative positivity rate where respondents agreed or strongly agreed on the Usefulness of CDSS-
integrated mobile was 80.43% (Mean Likert Score 4.03; 95% CI: 2.38– 4.77) with statistically significant
responses to the following questions on usefulness (p <0.05;one-sample t-Test) with the positivity rate
and mean Likert Scores as recorded below:
It gives me more information on what to advise patients: 97.83 % and 4.59(95% CI:4.40– 4.77)
It helps me remember all diagnostic procedures to be advised: 95.65%and 4.52(95% CI:4.33–4.72)
This tool helps me administer the right treatment and the right drug at the right dose: 93.48% and
4.48(95% CI:4.25– 4.70)
It enables me to determine which patients should be referred: 91.30% and 4.30(95% CI:4.12– 4.49)
I am more confident when I talk to patients about their conditions and recommendations: 86.96%
and 4.33(95% CI:4.1–4.55)
It is useful in doing my daily tasks: 82.61%and4.20(95% CI:3.98– 4.41)
By using this application, my work skills have increased, and I am able to accomplish my tasks
quickly: 73.91% and 4.09(95% CI: 3.81 – 4.37)
Two questions on Usefulness had statistically insignificant responses (p >0.05;one-sample t-Test) with the
positivity rate and mean Likert scores as follows:
The quality of my interactions has reduced due to the time spent in entering data and reading
instructions from the tool: 58.70% and 2.80 (95% CI: 2.38 – 3.23)
I often need to override the suggestions made by the tool: 43.48% 2.96 (95% CI: 2.53 – 3.38)
In addition, there was a statistically significant variance between different groups of health functionaries for
the question ―I often need to override the suggestions made by the tool‖ (p <0.05; One-way ANOVA)
with mean Likert Scores for four groups, showing a higher negative rate amongst ASHAs compared to the
other groups:
ASHA: 2.25 (95% CI: 1.63 – 2.87)
ANM: 3.57 (95% CI: 2.8 – 4.35)
Medical Officer: 3.20 (95% CI: 2.16 – 4.24)
Staff Nurse: 3.57 (95% CI: 2.17 – 4.97)
These issues were further delved into during the FGD sessions to gain insights into when and why the users
felt the need to override the suggestions made by the tool. During the facilitation of app use and shadowing
of case interactions, we had noticed that many ASHAs and ANMs had some difficulty in correctly
interpreting an indirect or negative question. The response to these questions may have similarly got affected.
There was no statistically significant variance between different groups of ASHAs, ANMs, Staff Nurses and
Medical officers for the other questions on usability.
287887/2021/DM&A
510 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
30
Figure 7: Usefulness Survey Results
Figure 8: Usefulness Survey Stacked % Results
9
7
3
8
1
1
1
2
14
8
9
2
5
1
1
4
5
21
15
19
16
18
17
16
24
10
17
19
8
29
22
27
27
18
10
0 51015202530354045
It is useful in doing my daily tasks
By using this application, my work skills have
increased, and I am able to accomplish my tasks quickly
The quality of my interactions has reduced due to the time
spent in entering data and reading instructions from the
tool
It gives me more information on what to advise patients
I am more confident when I talk to patients about their
conditions and recommendations
It helps me remember all diagnostic procedures to be
advised
This tool helps me administer the right treatment and the
right drug at the right dose
It enables me to determine which patients should be
referred
I often need to override the suggestions made by the tool
Usefulness (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
0%
20%
40%
60%
80%
100%
Usefulness % (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
287887/2021/DM&A
511 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
31
Overall Satisfaction
The cumulative positivity rate where respondents agreed or strongly agreed on the Overall Satisfaction with
of CDSS-integrated mobile was 76.09%(Mean Likert Score 4.10; 95% CI: 3.80 – 4.39) with statistically
significant responses to the two constituent questions(I would like to continue using this tool for my daily routine
and work AND Overall, I am satisfied with the CDS tool)(p <0.05;one-sample t-Test).
There was a statistically significant variance between different groups on the above two questions relating to
the overall satisfaction (p <0.05; One-way ANOVA) with mean Likert Scores for four groups, showing a
higher positive rate amongst ASHAs and Staff Nurses compared to the other groups:
I would like to continue using this tool for my daily routine and work: Overall Positivity rate
and mean Likert Score:73.91% 4.04 (95% CI 3.80 – 4.29)
o ASHA: Positivity rate: 100%4.50(95% CI:4.26 – 4.74)
o Staff Nurse: Positivity rate: 57% 4.29 (95% CI:3.41 – 5.17)
o ANM: Positivity rate: 50% 3.50(95% CI:3.06 – 3.94)
o Medical Officer: Positivity Rate: 40% 3.40(95% CI: 2.72 – 4.08)
Overall, I am satisfied with the CDS tool: Overall positivity rate: 78.26% 4.15 (95% CI 3.91 –
4.39)
o ASHA: Positivity Rate: 100% 4.55(95% CI:4.31 – 4.79)
o Staff Nurse: Positivity Rate: 85.71% 4.43 (95% CI: 3.38 – 5.48)
o Medical Officer: Positivity rate: 60% 3.60 (95% CI: 2.92 – 4.28)
o ANM: Positivity rate: 50% 3.64(95% CI:3.21 – 4.07)
Figure 9: Overall Satisfaction Survey Results
It was intriguing that the proportion of users who were over all satisfied was higher than those who would
like to continue using the tool. We delved into the reasons thereof in the FGDs.
Correlation
We were aware of the possibility of possibly receiving some random responses instead of a thought out
answer from the users. We therefore tried to find correlation between the responses to various questions and
see whether the correlations were logical.Certain responses to questions on Usability, Usefulness and Overall
Satisfaction had statistically significant correlations (p <0.05; Spearman‘s Rank Correlation test).
“It integrates/fits easily into my daily routine” had a strong positive correlation with “I would like to
continue using this tool for my daily routine and work”(ρ=0.70)
“It integrates/fits easily into my daily routine” had a strong positive correlation with “Overall, I am satisfied
with the CDS tool” (ρ=0.65)
0
5
10
15
20
Iwouldliketocontinueusingthistoolformy
dailyroutineandwork
Overall,IamsatisfiedwiththeCDStool
11
11
9
19
18
15
18
Overall Satisfaction (N=46)
Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
287887/2021/DM&A
512 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
32
“By using this application, my work skills have increased, and I am able to accomplish my tasks quickly”had a
strong positive correlation with“I would like to continue using this tool for my daily routine and work”(ρ =
0.62)
“I can accomplish my tasks more easily quickly”had a strong positive correlation with ―Overall, I am
satisfied with the CDS tool‖(ρ= 0.61)
“I often need to override the suggestions made by the tool”had a moderate negative correlation with“By using
this application, my work skills have increased, and I am able to accomplish my tasks quickly”(ρ =-0.50)
“I often need to override the suggestions made by the tool” had a moderate negative correlation with “It
integrates/fits easily into my daily routine(ρ =-0.42)
6.1.3.2. STG Adherence
Two quantitative methods of STG adherence that were adopted included a cumulative STG adherence trend
based on an analysis of completed journeys and specific STG adherence indicators for each of the 10 clinical
conditions included in the study.
Cumulative STG adherence trend (based on proportion of completed CDSS journeys)
Cumulative STG adherencerefers to a measure of the overall use of CDSS, regardless of the specific
STGs content that was processed by the CDSS engine. When a user completes a CDSS journey then
all necessary enquiries, decisions and recommendations that are part of the transformed content
associated with the merged STGs is completed. Abandoning a journey midway may be considered as
non-adherence to STGs.
From 21 October 2020 through 8 April 2021, 4460 beneficiary interactions had a clinical decision
support session started via the mobile app. Of these, 3995(90%) of the CDSS journeys started were
completed by either an ASHA, ANM, nurse or MO. CDSS journeys were not completed in 10% of
cases. At the start of the study, many journeys were incomplete possibly due to user hesitation, the
learning curve and technical reasons like poor or no internet connection, or software bugs, etc but as
the study progressed the numbers of completed journeys increased. The issues were monitored in
real-time by the Project Team to facilitate prompt intervention, which resulted in more journeys
being completed once the initial technical hitches (prior to December 2020) were sorted (Fig 10).
Figure 10: STG Adherence Trend for completed journeys(Oct-2020 to April 2021)
19-Oct26-Oct02-Nov09-Nov16-Nov23-Nov30 N0v07-Dec14-Dec21-Dec28-Dec04-Jan11-Jan18-Jan25-Jan01-Feb08-Feb15-Feb22-Feb01-Mar08-Mar15-Mar22-Mar29-Mar05-Apr
Journeys Completed117 8 1 0 7623995214825748534527214543257550834614445267118
Journeys Not completed113 8 3 0 84870304013132629121523403017115 2 6 2
Total Journeys 2210164 015110109125254957051137428416045561553836315550287720
% Completion of Journeys50%70%50%25%0%47%56%36%76%84%86%81%95%92%96%91%95%93%94%95%93%90%93%92%90%
50%
70%
25%
47%
56%
36%
76%
86%
81%
92%
96%
91%
95%
90%
92%
0%
25%
50%
75%
100%
0
200
400
600
Primary
Axis
-
Number
of
Journeys
Secondary
Axis
-
%
Completion
of
Journeys
Stable App release
Stable App releaseCOVID-19
vaccination drive
Early App release
One to one
retraining
287887/2021/DM&A
513 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
33
Adherence to Specific STG Adherence Indicators
A more detailed analysis of adherence to specific STG indicators was also carried out from analyzable
journeys in the data-set where all the information could be accessed.
Tables 7and 8providesthe STG adherence indicator clinical measures that werecomputed through SQL
queries on the Arezzo data base.
Summary of analysis:
STG adherence for maternal indicators was 77.28% (95% CI: 65.0 – 89.6) with statistically
significant mean adherence (p<0.05; one sample t-Test).
STG adherence for paediatric indicators was 74.96% (95% CI: 58.1 – 91.8) with statistically
significant mean adherence (p<0.05; one sample t-Test).
Interpretation of STG Adherence measures:
Majority of the indicators had 100% adherence. However, there were indicators that had less than
5% adherence. Majority of these were a consequence of either practice gaps at the ground level or
due to a lack of resources. E.g. Blood sample collection for blood grouping was 3%, but Urine
Samples were regularly collected for urine proteins (87%) and sugar (83%), due to entrenched
practices followed by the ANMs. Two consecutive BP readings are not taken four hours or more
apart if SBP > 140 mmHg or DBP >90 mmHg, again indicating a practice gap.
Another instance of practice gap in paediatrics is weight and temperature measurement on every visit
which is as low as 16%.
Screening for gestational diabetes mellitus was as low as 1% with zero OGTTs done for screen-
positive women. This has a two-fold causation – firstly, as per the medical officers and nurses, there
are virtually no cases of GDM in Bahraich, but interestingly there were not dipstick tests or OGTT
packs available for screening, which could also signal some level of undetected GDM at the study
site.
Another prominent indicator with low adherence was ―Documented referral when indicated‖, which
corroborates with the on-ground findings that indicated low referrals. The research officers did try to
do a root-cause analysis of this finding but the analysis was not able to be performed in real time, and
thus some of these cases may have been missed. This is an issue that will need to be reviewed in
more detail in future studies.
In Pediatrics, earlier a single developmental delay was triggering a referral with a very low threshold,
and it should have been set to only a high-risk level. This was subsequently fixed in the content and
unindicated referrals reduced significantly. However, this did impact the overall STG adherence
indicator related to referrals.
Details of secondary data analysis over the month of February of the STG adherence is contained in
Annexure 6.
287887/2021/DM&A
514 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
34
Table 7: STG Adherence Indicators for Maternal Health
* Eligible visits refer to the total number of visits that would have required STG adherence for each indicator. Not
all visits required STG adherence for every indicator, depending on the STG recommendation and the role. For
example, ASHAs were not expected to take blood pressure recordings on any visit. This is why the number of
eligible visits varies throughout the Table.
** Marked as green because not all women were able to provide an LMP but the question was always asked.
*** 14 referral recommendations were not followed because: user overrode the recommendation (4); family
reasons (4); transport issues (1); other reasons – not specified (9). Antenatal care (ANC) for uncomplicated pregnancy
1Documented LMP **1042926 89%
2Documented EDD10421042 100%
3History taken (specific symptoms, systemic illness, etc) 10421042 100%
4Weight measured930930 100%
5Blood pressure taken930930 100%
6Blood sample taken (for Hb)889829 93%
7Blood sample taken (for blood group, Rh)68318 3%
8Blood sample taken (for VDRL)889710 80%
9Urine sample taken (for protein)951827 87%
10Urine sample taken (for glucose)889741 83%
11Abdomen examined930930 100%
12IFA supplements prescribed1012967 96%
13Tetanus Toxoid prescribed766748 98%
14Calcium supplements prescribed889872 98%
15Anthelmintics prescribed897433 48%
16Counselling done (dietary advice)11591154 100%
17Counselling done (contraception)774702 91%
18Counselling done (violence)775773 100%
19Counselling done (rest)11591152 99%
20Referral recommended ***3319*** 58%
Gestational diabetes mellitus (GDM)
21Screening for GDM (blood glucose)88915 2%
22History taken for signs & symptoms of hyperglycaemia 13431343 100%
23OGTT done for screen positive women00 0%
Hypertension in pregnancy including eclampsia
24
History asked (hypertension or pre-eclampsia/eclampsia in any
pregnancy or family history)
719719 100%
25
Two consecutive BP readings taken four hours or more apart if SBP ≥
140mmHg or DBP ≥ 90mmHg
200 0%
26Test for urine albumin in case of high BP2014 70%
Anaemia in pregnancy
27Documented history suggestive of anaemia (pallor, palpitations, fatigue)13431343 100%
28Haemoglobin test at each visit889829 93%
29IFA supplements – one tablet twice a day37176 20%
Postnatal care
30Iron prescribed138136 99%
31Contraceptive advice given124121 98%
32Nutrition advice given124124 100%
33Breast feeding advice given124123 99%
No.STG Adherence IndicatorEligible visits*
No. of visits with
evidence of
compliance
Adherence (%)
287887/2021/DM&A
515 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
35
Table 8: STG Adherence Indicators for Child Health
* Eligible visits refers to the total number of visits that would have required STG adherence for each
indicator. Not all visits required STG adherence for every indicator, depending on the STG recommendation
and the role. This is why the number of eligible visits varies throughout the Table above.
6.1.3.3. Qualitative data: Focused Group Discussions
Following the quantitative surveys, it was deemed critical to uncover latent insights on the usability,
usefulness, and propensity to facilitate adherence to STGs with CDSS through FGDs, that could further
enhance the learnings from the results obtained thus far. All study participants were divided into five groups
based on their role and district for five virtual FGD sessions. Each FGD usually had the members from a
particular cadre so that there was an ease among the participants due to familiarity and no fear of a superior
officer‘s presence. The health functionaries met at a convenient time and place of their choice near their work
area. All sessions were attended virtually by the researchers while the project implementation partner
representatives present on-site facilitated the meeting through virtual platforms. Although these FGDs
externally transpired as informal discussions, the PIs leading the sessions followed a tacit structure that
involved broad questions that further led into more specific areas of enquiry, with equal representation from
all participants. There was a pre-decided checklist used by the investigators, which largely focused on the
same questions as the quantitative survey but intended to unravel the reasoning behind the responses. The
sessions were recorded with informed consent on the assurance of anonymity of individual responses.
The responses received in the meetings were broadly classified around the themes of usefulness and usability
of the tool and have been organised thematically, on the same lines, below. We have recorded the verbatim
comments in Hindi and also translated them into English for easy reckoning. Feeding
34 Breast feeding within one hour of birth34 34 100%
35
Counselling done on lactation support, breast feeding, IYCF and KMC (as part of LSU requirements), on
complementary feeding and on feeding and recommendations during sickness and health
2569 2569 100%
Immunization
36 Documented immunization history as per National Immunisation Schedule (NIS) 2893 2893 100%
37 Recommendations for vaccination as per NIS1799 1799 100%
Pneumonia
38 Documented assessment (signs of pneumonia and pneumonia severity)2798 2798 100%
39 Documented referral when indicated16 5 31%
Diarrhoea
40 Documented history (feeding, SAM, diarrhoea, blood in stool)3080 3080 100%
41 Documented assessment (persistent diarrhoea, dysentery, signs of dehydration) 3094 3094 100%
42 ORS packet prescribed when indicated26 26 100%
43 Zinc prescribed when indicated26 25 96%
44 Documented referral when indicated7 2 29%
45 Antibiotics prescribed 47 12 26%
Care of the sick neonate
46 Documented history (preterm, LBW, intrapartum/post-partum complications)356 356 100%
47 Documented assessment (ETAT)62 62 100%
48
Documented assessment (feeding, crying, breathing, icterus, pallor, cyanosis, abdominal distension, neck
rigidity)
356 356 100%
49 Documented referral when indicated80 31 39%
General
50 Documented birth weight179 128 72%
51 Weight measurement on every visit3041 492 16%
52 Temperature measurement3161 521 16%
No.STG Adherence Indicator
Eligible
visits*
No. of visits with
evidence of
compliance
Adherence
(%)
287887/2021/DM&A
516 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
36
Usefulness of the tool
We asked the participants as to what was the best thing they liked about the tool to document what were the
most useful features from their viewpoint. The responses are documented below clubbed under common
thematic domains.
It was evident in all the FDGs across all sections of the users that the foremost perceived
benefit was a sense of empowerment due to improved knowledge.
ASHAs and ANMs commonly supported a general perception of augmentation of knowledge as
stated by two of the participants:
o ―ज़्यादाजानकारीममलीींहै‖(Got more information) and ―App सेकाफी knowledge ममलतीहै”(The app provides
a lot of knowledge),
o more specific elements of knowledge, as mentioned by another participant
―बहुतसारीजानकारीजोहमेंनहीींथीवहइस app के ज़ररएमालूमहोतीहै, जैसेकीब�ेमजसतरहबढ़तेहै,
मकसउम्रमेंउन्हेंबोलनाहै, चलनाहै, जोइसके Questions मेंआताहै, वहहमेंपहलेनहीींपताथा(A lot of information that
we were earlier not aware of, we got to know through this app, such as the way children grow, at what age
they should talk, walk, which comes in the form of questions, that we did not know before).
o There was also a sense of incremental knowledge compared to theearlier state, as suggested
by one participant ―जोपहलेपूछतेथे, उससेextra हैइस app में (This app has extra information than what
we used to ask earlier).
There was a manifest inclination towards the capabilities of the app to serve as an effective
job aide, particularly by ASHAs and ANMs.
o Automatic calculations of dates: One participant mentioned ―गभभवतीममहलाकी LMP से EDD
कापतालगजाताहै (We get to know a pregnant woman's EDD based on her LMP). ―काफीसारीcalculations
अपनेआपहोींजातीहैं” (Quite a few calculations get done automatically).
―इसमेंहमटीकालगानेजातेहैतोअगलीताररकआजातीहै” (If we go for vaccination this gives you the next date) said
another.
o Tips for situation specific and appropriate counselling: The app‘s capabilities to generate
sound counseling and referral advice was also appreciated.
―App जोसलाहबताताहैवोबहुतअ�ाहै (The advice that the app give is very good).
―App मेंजो prompt आतेहैंउसकीवजहसेमुझेज्यादासोचनानहीींपडता. App अपनामदमागलगाताहैं (The prompts
that come through the app ensure that I don’t need to think too much. The app uses its brain).
o A significant insight on the clinical impact of CDSS was that latent High-Risk Pregnancies
(HRP) were being uncovered.
One participant mentioned ―इससेहमेयेपताचऱजाताहैकि HRP हैकिनही‖ (With this we get to know
whether it is an HRP or not).
o Identification of high-risk pregnancy or a sick child ―Tab
ज्यादाजानकारीपूछताहै, हमेंपताचलजाताहैमकसको refer करनाहै, मकसकोनहीींकरना‖ (The tab asks for more
information and we get to know whom to refer and whom not to).
―हमारे काममेंबहुतआसानीहोगईहै. Referral cases पताचलजातेहै. बहुतसारे suggestions हमेममलतेहै. (Our job
becomes much easier. We get to identify referral cases and get a lot of suggestions).
Another participant mentioned ―हम HRP गभभवतीकोबताभीदेतेहैमकवजनकम हैं, खूनकमहै,
चलोहमआपकोमदखालातेहैं‖ (We tell HRP cases that your weight is less, or you have anemia, let me
get you checked).
o The Clinical Decision Support capabilities were lauded by the medical officers and nurses as
well.
One of the MOs mentioned ―इसमेकाफीऐसीचीजेंहैजोहमclinical practice मेंmiss करजातेहै (There are
quite a few things in this app that we usually miss in our clinical practice). A staff nurse said
287887/2021/DM&A
517 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
37
―जोचीजहमनहीींसोचपातेवोतुरीं तहमे suggest होजातीहै (There are things that we don’t think of, but that get
promptly suggested to us).
Another medical officer mentioned ―Paediatriciansकोसाराकु छयादरखनामुश्किलहोताहैं. Appमें age
के महसाबसेसाराकु छआजाताहै- treatment क्यादेनीहै, counseling क्याकरनीहै, diet advice येसाराकु छअ�ाहै
(It is difficult for Paediatricians to remember everything. The app provides everything based on the age - what
treatment to give, what counseling to provide, diet advice – all of this is good).
―Patient की history, symptoms, मकसे refer करनाहै CDSS सेपताचलजाताहै” (Patient’s history,
symptoms, whom to refer to – all of this we get to know from CDSS) said another MO.
The ability to automatically generate clinical summaries also seemed to resonate with the
participants. They appreciated the fact that a readily organized complete summary is available to
them at the end of interaction (for first responders) or when they receive a referral (staff nurses and
MOs)
o ―Summary मेंजोपूराrecord आजाताहै, उसमेसारीजानकारीहमेएकसाथममलजातीहै,
वोदेखके लगताहैकीवाकईमेंअ�ाहै. (The complete record comes as part of the summary and we get all the
information at once – looking at that makes one feel that this is really good).
o ―लेखाजोखासेफायदाहोताहै, कमसमयमेअपनाकरके लाभाथीकोभीसमझादेतेहै‖ (The clinical summary is beneficial,
we do our jobs in less time and counsel the beneficiaries as well).
o ―अपनेआप साराकु छ summarize होजाताहै. कु छही point हमे add on करनेपडतेहै, वनाभसाराकु छ summary
मेंआजाताहै (This automatically summarizes everything, and we need to add very few points, otherwise
everything comes in the summary).
Some of the participants, particularly at CHC, felt that it helped in an effective continuum of
care and possibility of faster care for the high risk / referred cases
o One medical officer mentioned ―पहऱेहमेसारीhistory ऱेनीपड़तीथी, अभीहमज्यादातरtreatment
part हीिरतेहै‖ (Earlier we had to take the entire history, but now we mostly focus on the
treatment part).
o One medical officer gave a clear description of the impact ―HRP load बढ़जायेगा,
परयहअ�ाहीहै. हमारे पासमफलहालANC case third trimester मेंआतेहैऔरउनकी screening
अगरASHA ANM करे तो HRP काशु�सेहीgrassroot identification होजाएगाऔरउनकाreferral and
management अ�े सेहोजाएगा, cases improve होजाएीं गे. (The HRP load has increased, however this
is good, because we currently get ANCs in the third trimester and if their screening is done by ASHAs
or ANMs, a possible HRP can be detected early with grassroot identification and their referral and
management will be better and cases will improve)
o Patient जबहमारे पासअचानकआताहैतोपताचलजाताहैमकहमनेउसेपहलेदेखाहैमकनही, या ASHA, ANM
देखचुकीहैमकनहीीं. (If a patient comes to us suddenly, we immediately get to know if we’ve seen the patient
earlier, or if the ASHA or ANM have seen the patient earlier) and we do not have to repeat the
whole history. “हमज़्यादाकामहोनेके बावजूदरेफ्रे डमरीज़कोदेखसकतेहै”We are able to help the referred
case appropriately despite our otherwise busy schedule,‖ said a staff nurse who immediately
thereafter had to rush to conduct a delivery.
There was also an overall sense of satisfaction with CDSSthat could be extrapolated from
comments such as ―हमेंमवश्वासहैइसपर‖ (We have trust in this), ―अभीहमइससेसींतुष्टहै‖ (We are now satisfied with
this) and ―App आगेभीचलेगातोज्ञानममलेगा‖ (If the app continues in the future, we will get knowledge).
Usability of the tool
As almost all the participants felt that the tool was useful to them in more than one ways, we wanted to know
why did it not reflect as emphatically in their choice for continued usage of the tool. What were the
hindrances and challenges which are not making it universally appealing?
287887/2021/DM&A
518 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
38
The biggest concern raised by several participants was the amount of time it takes for data
entry.
o One participant mentioned ―जबहमफीमडींगकरतेहैतोबहुतसीमदक्कतेआतीहै, जल्दीहोताभीनहीींहै.
एकएकमरीज़के फीमडींगमेंकरीबआधाघींटालगजाताहैकभीकभी‖(When we enter data there are a lot of
challenges, it does not happen quickly, it takes almost half-an-hour sometimes to enter data for one
patient).
On further discussions, it was clear that the longer time taken was due to several reasons,
many not directlydue to the current interaction, e.g.for the creation of the initial registration to
create a family record; due to the initial teething troubles and learning curve. This was also
reflected in the quantitative data as fall in incomplete journeys over time as discussed above (Fig 10).
o as one participant mentioned ―Register करके journey करनेमेंज्यादा time लगताहै, परअगली visit
मेंआसानीहोजातीहै‖ (It takes a lot more time for registration and then to complete a journey, but in the
next visit it becomes much easier).
o ―Starting हैइसमलएकु छissues लगरहेहै. करतेरहेंगेतोआसानहोजाएगा‖ (It is just starting which is why we
feel there are issues. If we keep doing it, it will get easier).
Another reason cited for more time required for the interaction using the tool was perhaps due to
inability to skip any part of detailed history taking while using app. quite unlike prevalent direct
interactions which are very pointed and limited like a firefighting approach.
o ―बहुतज्यादाचीज़ेपूछनीपड़तीहैं।कभीकभीतोबेनेमफशरीके हताहैक्यादीदीइतनासाराक्यूूँजाननाहैहमकोदेरहोरहीहै।(
We need to asktoo much in detail. Sometimes the beneficiaries tell us “Why do you need to ask so much.
We are getting late”).
However, when we asked which areas were unnecessary and can be trimmed, there were no
suggestions and often the group agreed that its lengthy but is useful.
o अ�ाहै, सबकामकीबातहैंबसहमअभीतकपूछतेहीनहीींथे।(It is good the way it is because it ispertinent, it is
just that we were not asking all this information till now).
It also become evident that the technology introduction was a disruptor in several ways. Currently
the interaction with beneficiaries is somewhat limited and the expectation of the beneficiary is to get
free vaccination or drugs and a quick turnaround. The time investment to create a detailed holistic
health record by the provider and the beneficiary has not been made in the past. The present tool
asks for more detailed and meaningful interaction to which both the stakeholders are not oriented.
o It is also challenging when multiple beneficiaries come together, as one participant
mentioned. ―Beneficiariesहमारेमहसाबसेनहीींआतेवोपूराझुींडबनाकेआतेहै. वहाींएकएककी journey
करनाimpossible होजाताहै. (Beneficiaries don’t come as per our schedule, they sometimes come as a
group, when completing a journey for each one of them is impossible).
o The beneficiaries are not willing / able to give time for the complete detailed history.
―लाभाथीसमयनहीींदेपातेहै, जल्दीमेंहोतेहै‖ (Beneficiaries are not able to give time, they are in a hurry).
o Moreover, it is currently not possible to see multiple patients together since the
app allows only one journey at a time. ―जबतकएक journey पूरीनाहोदू सरीशु�नहीींकरसकते
(Unless you complete one journey, you cannot start the next).
Using of a smart device for collecting data may be misinterpreted due to lack of awareness, lamented
a MO.
o ―Public ऐसीहैमकवहकहेंगे Doctor साहबmobile परखेलरहेहै, हमेनहीींदेखरहेहै” (The public here is such
that they would say the Doctor is playing on his mobile and not seeing us).
Another underlying concern which affected the choice of continuing adoption of the tool, particularly
among ASHAs, was the fear that it will increase their work load by increased identification of sicker
cases. Interestingly, in one of the FGDs, a very motivated and champion user wished to retract her
emphatic statement on the benefits of CDSS as it had made her co-workers unhappy for they felt she
was asking for a change that is going to increase their workload.
287887/2021/DM&A
519 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
39
o “औरज़्यादाबेनेमफशरीजकोसेंटरलेकरजानापड़ेगा‖ (We will need to accompany many more to the health
centers).
Thequantitativesurvey hadbrought about some concerns about the referrals being generated by the tool
as some of the recommendations for referrals were perceived to be unnecessary. However, the
participants were emphatic in stating across FGDs that they never had a situation where the app had
not suggested a referral when the workers felt it was required.The users agreed that it efficiently
identified all probable referral cases.
o ―कभीकभीapp refer करनेकोबोलताहैपरहमारीसमझमेrefer करनाज�रीनहीींहोता, परइसकाउल्टाकभीींनहीींहुआहै‖
(Sometimes the app asks us to refer but we don’t think referral is necessary, however the vice-versa has never
happened).
Discussion about possible facilitators, challenges and hindrances to adoption of CDSS tool brought
some more interesting insights
The reluctance to use the tool in part was because the users thought that it will be an additional
method and not a replacement of the physical mode as was clearly brought up at one of the
FGDs,involving 10 ASHA workers. When asked how many would like to continue using the app, 6
raised their hands. The four that did not, mentioned the following reasons for their reluctance: limited
time available and demanding work, lack of adequate incentives/payments, and the additional work
pressures during COVID-19. However, when asked that if the application were to replace the
handwritten ASHA register and other reporting requirements, all 10 ASHAs unanimously said they would
find it much easier to adopt. This was echoed by others too:
o One participant mentioned―डायरीवालाकामइसमेहोजाएतोबहुतअ�ाहै‖ (If the job of diary (daily
registers) can be done by this app, it will be very good).
o ―अभीदोदोकामहैं. अगर ASHA register कीजगहमसफभ येapp आजाएतोअ�ाहोगा,
क्योींमकबारबारभरनानहीपडेगा‖ (Currently we have to do two things. If the ASHA register gets replace by
this app, it will be good, and we won’t have to duplicate our work).
o ―OPD मेंdaily register भरनाकमहोजाएगा, हमारे पासएक handy app होजाएगाऔरबहुतआसानहोजाएगा‖
(If the need for filling out the OPD daily register gets reduced and if we have a handy app to do this, it
will become much easier).
However, there was a concern with a fully digital mode not being fully reliable
o One participant, who was an early adopter and had done a lot of cases, mentioned
―साराdigital होजाएये possible नहीींहै, थोड़ातो manual रहनाचामहए, अगरapp नाचलेतो” (It is not possible
that everything goes digital. Some of it should be manual – what if the app doesn’t work). (we will
have no backup).
The issue of incentives was also raised by some participants
o ―ज्यादासमयमनकालनापडताहै. इसकामके मलएहमेकोईअलगसे payment भीनहीींममलताहै(We need to take
out extra time for this and do not even get paid for the additional work).
But when we reminded them that the app is not adding any new work rather easing their work through in
a better and efficient method, it evoked mixed reactions from acceptance to need for better payment or
incentives as they felt that they were poorly paid.
Suggestions for improvement to improve usefulness and usability
Several participants had inputs on how to improve the app further. Although many of these suggestions were
predominantly around the app workflow, and not directly related to the CDSS, it was felt that these
suggestions, if incorporated, would increase the overall value, and consequently, the adoption of CDSS.
Internet connectivity emerged to be a critical bottleneck in adoption of CDSS and there was
a clear preference for an “offline” mode.
287887/2021/DM&A
520 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
40
o One participant mentioned ―सबसेबडी problem network कीहैं‖ (The biggest problem is that of the
network). Another suggested ―मबना internet केapp चलजाए to ज्यादाफायदाहोगा‖ (If this works without
the internet it will be more beneficial).
The ability to correct data entry errors was perceived to be a critical missing feature of the
product.
o ―गलतentry कोcorrect करपाए‖ (We should be able to correct wrong entries).
o ―मकसीलाभाथीका data feed करदेतेहैऔरवोहमेमकसीगलतीकीवजहसेchange
करनापडेतोहमनहीींकरपातेहैंऔरमफरसेहमे feeding करनीपड़तीहै‖ (If we make a mistake while feeding data
and need to change we’re not able to do this and have to start all over again).
o ―CDSS मे next औरbackका system होनाचामहए. कभीगलतीसेकु छ submit करे तोवो back
नहीींहोपाता‖(CDSS should have a system for going back and next. If we submit something by mistake, we
aren’t able to go back). नएमसरे सेकरनापड़ताहै. We have to restart the journey.
While the investigators agreed with the need but had concerns about the risk of data fudging by
enabling re-entries.
A recurring and consistent suggestion was that of capability to generate due lists, missed
appointment alerts, area or worker-wise list, an optimized workflow, and notifications to the
nurse or doctor on when the beneficiary is scheduled to be seen by them.
o ―Due List add होसके तोउससेकाफीसुमवधाहोजाएगी / ब�ोींके टीके की due list आनीचामहए‖ (If a due list can
be added, it will be very useful, a due list for pediatric vaccination should come).
o The need for a PNC due list was expressed clearly
―मडलीवरीहुईऔरहमप्रसवोत्तरमाहमेंफीमडींगकरनेचलेगएतोहमेंमदखाईनहीींदेता, तोहम CDS कै सेकरे(If a delivery
happens and the woman becomes postnatal, then we can’t seethe patient details in app so how can we do the
CDS?).
o “App ने indicate करनाचामहएकीकोइ visit छू टगई‖ (The app should indicate that a visit has been missed).
o ―गभभवतीममहला delivery के बादअपनेआप lactating mothers
कमलस्टमेंचलीजाएतोहमाराकामआसानहोजायेगा‖(After delivery if the mother gets added to the lactating mothers
list, it will become much easier).
o ―OPD में photo लेकरहमरातमे journey complete करतेहै. Time slot होऔर referral list
होतोअ�े सेकरपाएीं गे‖(We take photographs of the case record and complete journeys at night (because of the
rush in VHND). We will be able to do this better if there are time slots and referral lists).
o ―Referral मेंoption होनाचामहएकीहमकहा refer कररहेहै, मरीजकोकै सेभेजरहेहै, औरउनको alert करपाए. जैसे
CHC पेrefer करोतो CHC पेहीजारहाहो, मकस doctor कोrefer कररहेहै. जानकारीसबकोहोनीचामहएऔर tab
सबके पासहोनाचामहए‖ (There should be options in referral – where are we referring, how are we sending the
patient, and alerts. E.g. if we are referring to the CHC, the patient should go to a particular CHC and to a
specific doctor. Everyone should have the information, and all should have the tab).
o ―VHNDमे patient Register करकेjourney करनामुश्किलहै. हमेपतानहीींचलतामकASHAनेVHND
के मलएpatient Register मकयाकीनहीीं. कभीकबारdouble registration भीहोजाताहै‖ (It is difficult to register
patients during busy VHNDs. We don’t get to know whether the ASHA has registered the patient for
VHND. Sometimes there is duplicate registration).
o ―Journey next day करनेकीसुमवधाहोनीचामहए, कु छमदक्कतहोतोलाभाथीक specially बुलाकरadvice देपाए‖
(There should be a facility to complete a journey on the next day. If there is a problem, we should be able to
specially call the beneficiary and give them advice).
Medical Officers suggested some improvements in clinical content.
o One MO mentioned ―इसमें NCDs add होनीचामहए‖ (This should also cover Non communicable
diseases)
o Another MO suggested ―इसमेdifferential diagnosis का hint होनाचामहए, मजससेहमारीpersonal
knowledge में improvement होसके” (This should have hints on differential diagnosis, so that there is
improvement in our personal knowledge).
o “थोडी language clear होनीचामहए. एक answer के दोअथभनामनकले” (The language should be clear. One answer
should not have two meanings).
There were concerns about the multiplicity of mobile apps, particularly the ANMs.. One
respondent mentioned that there were too many apps being introduced. The Government is also
287887/2021/DM&A
521 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
41
releasing the ANMOL app which they are required to use. If all of them get integrated into one
simple app, it will be much easier to use.
7. Discussion
This feasibility study to assess the usefulness, usabilityand adherence to STGs, with the use of CDSS in
Indian public healthcare settings generated significant insights into the pre-study rationale, beyond
conclusively proving the research hypotheses.
Transformation of clinical guidelines to a digital interactive smart tool
Feasibility of transformation of local guidelines and developing smart clinical decision support tools for use
by health functionaries in the community and primary care settings was firmly established. Even though this
pilot proof of concept study had worked only with few maternal and pediatric (under five yrs. age)conditions
considered important from public health viewpoint, yet it demonstrated the ability to convert these into a
reliable digital tool with complete fidelity to the written guidelines across various selected conditions and even
with situation where more than one of these co-existed. The users had a choice to use the tool either in Hindi
or English language.
Empowerment of Front-Line Healthcare Workers
There wasvast appreciation of the ability of the tool to usefully add value to interactions with the
beneficiariesby the users.As was evident from the surveys and FGDs, the CDSS intervention empowered
FHWs at every stage, with the augmentation of knowledge that came sequentially at the right point
in the flow in each screening journey, so it acted as a fantastic job aide for everyone. This came out
strongly from all the three cadres of the study participants. While ASHAs and ANMs remembered very less
of their pre-study guidelines training sessions, the app reduced such kind of memory attrition, because the
tool takes one through the whole process of evaluation and brings up all the relevant questions required for
an individual patient, and the FHWs do not have to struggle to find what guidelines apply. There was also a
strong sense of empowerment since the study extended the FHWs‟ horizon into some areas that they
had previously never worked on, like developmental milestones, which they‘d never used in the past
because the recently introduced MCP card was not a part of their routine.
Broadly, the benefits accrued are described here.Firstly, the knowledge of FHWs was augmented through
CDSS, based on reliable and extensive information gleaned from selected guidelines. Relevant advice within
the CDSS app was generated at the right time and strategically placed to be invoked when it was required at
the point-of-care. Normally they would have to refer to a guideline and look up static information, but the
CDSS app is designed in such a way that relevant clinical information came as part of the flow, at the right
place and at the right time, empowering FHWs with knowledge in context of the workflow. Secondly the
attitude of FHWs was also positively impacted through better structured advice. They felt more confident
in their interactions and appreciated the structured and detailed action-oriented advice relevant to the
caseproduced by the tool.Thirdly, the workers also appreciated that the tool was more comprehensiveand
reliableas it never missed a case which required referral. The structured summaries which were available
at the end of an interaction made the functioning efficient and complete. This critical point of impact was
made by several participants, particularly the MOs as they highlightedthe ability of the app to empower
ASHA‟s and ANM‟s to systematically screen, identify and timely refer high-risk cases particularly
HRPs in early pregnancy rather than in third trimester when it is too late. The MOs expressed a clear desire
to augment the CDSS app with additional elements like differential diagnosis and suggested
inclusion of conditions like non-communicable diseases (NCDs), indicating need for feature
augmentation for improved acceptability.
Task Shifting and provision of seamless continuum of care
Although the actual process of effective task shifting is a gradual process, there was a successful
demonstration of the potential of task shifting with the CDSS intervention as part of this study. Due to
287887/2021/DM&A
522 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
42
the thoroughness of the app in capturing an entire visit in the form of a lucid summary, by the time the
beneficiary reached the next level of care, a significant part of necessary clinical information was already
available, and it reduced duplication of effort that occurs each time a frontline worker interacts with a patient.
The app can also serve as a two-way communication tool, between ASHA and ANM for seeking
opinion and advice by sharing case summary and between ANM and MO, without the ANM
needing to accompany the patient to higher center. The ANM can send details that can be accessed by
the MO and initiate management under the supervision of the MO. This provides a unique opportunity of
continuum of care for an individual as long asthey are within the system. This also provides an opportunity
to seamlessly provide escalation and de-escalation of healthcare-related activities to any beneficiary.
As there was clear documentation of the interaction and the MO has the opportunity to verify the inputs
from ASHA/ ANM on a particular case, it provides an opportunity for on-the-job support and training to
these functionaries through feedback. The opportunities to analyze which healthcare workers need to be
trained on better health screening, based on information about practices available within the system, can
accommodate capacity-building efforts towards effective task shifting. The timely and contextual
cognitive guidance willmake theASHA or ANM more confident in taking on higher responsibilities with time.
Through the CDSS App, information capture has partially moved to the lower cadre and treatment has
moved to the upper cadre in the field, whereby medical officers just verify the history and assessments,
reconfirm and add on to the treatment, if required, resulting in an optimized workflow and conducive
grounds for task-shifting. In addition, this also provides credible information for supportive supervision
across the cadre.
The clinical impact of such task-shifting could be significant. Clear communication between ANM and MOs
can optimize referrals, and early detection of HRPs, when the ANM is able to send a list of HRP referrals
and the staff nurses as well as MOs at the centers exactly know which patients are expected and whom to
prioritize. This could also reduce waiting times for HRPs who would receive treatment and intervention
earlier. Further, follow-up care could also be shared with the MO in a two-way communication
model, allowing patients to be sent home on supervised domiciliary treatmentwith effective follow-up care by
ANM and ASHA. This, in some ways, is step-down care in the reverse continuum, which is also one aspect
of task shifting, economical and a patient and facility-friendly output. Some eligible high-risk patients can get
care at home without disrupting their family life, income, or affecting caregivers, by seeking care closer to
home and improving its acceptability. It will also allow fast turnover of inpatients in the higher centers like
CHC. Task shifting is not merely about a complex task being performed at a lower hierarchy, but it is also
sharing of tasks, which is extremely important from a public health point-of-view, especially in terms of
breaking the silos, that most systems are currently working in,thus making the care continuum more efficient.
Acceptability of CDSS
Usability and usefulness of CDSS were two primary outcome measures that inherently impact its acceptability
by FHWs. Some manifest benefits that accrued with CDSS included the optimization of the continuum of
care through better two-way communication between the ASHA, ANM, Staff Nurses and Medical Officers
based on the automatically generated clinical summaries obviating the need for personally accompanying
patients. Also, a longitudinal record grounded in health screening and clinical information ensures
continuity of care even if physical records like the Mother-Child Protection (MCP) card are misplaced.
Acritical insight was that workflow improvements in the operational part of the App will have a bearing
on the acceptability of CDSS since the operational and clinical elements of health screening go hand in
hand.The ability to integrate optimized workflows with CDSS recommendations is a powerful combination,
allowing for due-lists, alerts, and reminders along with evidence-based screening and recommendations. This
is do-able but was beyond the scope of the present proof of concept study which largely focussed on
establishing feasibility of the idea of adoption and conversion of the guidelines to a computer interpretable
format and creating a decision support system customised to the country guidelines.
Data Provenance
Arezzo
® demonstrated the ability to generate high-quality clinical data sets during usage as part of
the FHWs workflows during the study and this holds a lot of potential for further analysis and use in the
287887/2021/DM&A
523 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
43
future. This operational and clinical data can lend itself to post-hoc analytics leading to a secondary gain in
terms of an increase in district-level data about care delivery, referrals, and outcomes. It is also critical
to realize that while electronic data does get captured as part of the workflow, the eventual analytics
processes need to be thought out at the outset, specifically in terms of what reports, dashboards and
views are required, how the data is indexed and standardized. The granularity of this data can be from the
district-level to the PHC-level and further to the individual FHW-level, signifying a major potential for
supervision and real-time monitoring. It must be supported with a simplified information retrieval process
which is navigable evenby non-technical functionaries. The eventual goal would be to ensure that the desired
analytic views are available at the click-of-a-button with the ability to drill down to the individual FHW-level,
which is critical in terms of root cause analysis in terms of contributory factors such as deficiencies in
infrastructure, training and guide actions required. This will enable early identification of weak spots in
the system and weak links to direct focussed interventions to strengthen them instead of generalized
interventions for all. This couldresult in effective and real time course correction and enable the system to
reach predetermined goals efficiently without waiting for time and manpower consuming nationwide surveys
every 5-10 years. Also, the possibility of district-level dashboards linking outcome measures to region-
specific practices and training needs for FHWs is a significant public health opportunity. In fact, if used at
scale, it holds the potential to partially automate NFHS-like national surveys.
8. Lessons learnt and lateral insights for future studies
Lack of Harmonized Standard Treatment Guidelines
The biggest issue that the research team faced in adaptation of STGs was the lack of harmonized and
updated guidelines for most conditions. There were some variances observed between guidelines. Thus,
e.g., what the ASHA Module recommends was at variance from what the SAANS (Social Awareness and
Actions to Neutralize Pneumonia Successfully) program contained. The research team had to reach out to
several functionaries to get a sense of the expected updates to guideline since they were cautious of
introducing any element that was not part of existing guidelines and that could confuse health workers. A
solution to digitize guidelines does partially address this issue, primarily by identifying the underlying
fragmentation in order to institute corrective action. Another challenge faced was due to vertical health
programs that are focused on one problem, but when it comes to an individual patient there usually are
overlapping symptoms, and this can create a lot of confusion, especially with more than one terms describing
it across different guidelines. Thus, the need for harmonization of guidelines is also at the level of
guideline-specific-language. The PIs and ERG had to spend efforts tomerge multiple guidelines and
help the developingteam to create CDSS solution with the ability to render them seamlesslyto the FHWs,
and thus the Arezzo
® technology could facilitate the continuity of care required with comorbidities,
complications, and disease clusters.
Another critical issue faced by the research team was the presence of rather outdated guidelines. Maternal
guidelines from GOI have not been updated for quite some time now, and thus there were mismatches
between practices at the ground-level and published guidelines, especially for ASHAs and ANMs.
There were also some published guidelines that had not been implemented yet such as Anemia MuktBharat,
GDM guidelines, and the research team had to join those pieces together. While health is primarily a state
subject, there are vertical National Health Programs financed by the Central Government. But there are
variations at the State-level implementation of these programmes as the State can always adapt these
programmesby whatever they think is relevant to their own region.Also, some elements that are there in the
guidelines may not exist in the periphery, especially due to lack of resources. However, the starting point of
such a digital solution should be a common minimum agenda that is acceptable across different state
level health functionaries. It is important that the final product is not devised in purely top-down approach
and includes local adaptations for a confusion-less transition to digital mode. Importantly, theprocess of
digitization makes STGs more dynamic, improves the ability to harmonize various guidelines,
leading to faster dissemination and therefore, may be more efficient way for implementing changes
/ modifications to guidelines. Further the ability to capture practices in different areas enables uncovering
of gaps that can be addressed in trainings or updates, thus improving the process of implementation of
guidelines.
287887/2021/DM&A
524 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
44
As for the adaptation of STGs to CIGs,the process of content transformation to digitize guidelines was
quite challenging. Firstly, a change in the mode of delivery of guidelines into a format where the FHW
would screen beneficiaries with initial questions and then leading questions based on previous responses
required a thorough deconstruction followed by reconstruction of the STGs to align it in a conducive format.
Further it took time for the research team to become familiar with the content transformation processand for
the technology team to become more aware of locally relevant clinical practices. Furthermore, the need to
incorporate increasing levels of sophistication in the upstream hierarchy of the healthcare workers required a
cadre-wise distinction of various parts of STGs that did not always specify who had to execute what part of
the guideline and how one could cater to variable patient cohorts based on their socio-economic status and
level of education. This further complicated the scope of the content transformation process. Also, the level
of manual effort involved in making small changes to the CIGs was something that took time. However, the
delay in starting phase 2 due to COVID allowed a prolonged Phase 1 to the investigators to fine tune the
conversionprocess. In the initial phases, a small change would shake the whole journey so incorporating
changes in the app was a challenge and time-consuming task.
Local-language translations posed a significant challenge in this study. There was significant effort put
in by the research team for Hindi translations and the process of using spreadsheets to communicate these
translations to the technology team based in London (not familiar with Hindi) was not optimal leading to
several iterations and rework.Although the local-language at the study site was Hindi and most of the research
team was comfortable with Hindibut further changes had to be incorporated later in the implementation
phase based on the user feedback. This largely resulted from differences from the acceptable or
understandable vocabulary /locally used hindi words. Future scale-up efforts into other local languages will
require fool-proof translation processes with involvement of few well-versed local health functionaries early
on.
Lack of protected time or incentive for study participants, COVID pandemic related disruptions
Time and timing (occurrence of pandemic) were the biggest limiting factorsfor the study. The teams
were working across continents and the time differences impacted coordinationand added to delays despite all
efforts by the teams to find a time slot on a regular basis. Notwithstanding extended study period due to the
COVID-19 pandemic, local restrictions and reorientation of the health care entailed that the opportunities
were still limited for effectively evaluating workflow optimization. As ASHAs are notfull-time health
functionaries, absences due to local festivals, social reasons and health or confinement also impacted the
workflow.
Many homebased activities were suspended due to COVID hence related guidelines could not be evaluated in
the context of continuum of care visit by visit. It affected evaluation of interventions during follow-up visits
and the impact of the Home-Based New-born Care program. The study team could not stay in the field for
long time for non-participant observations of the interactions. Shadowing of interaction with the beneficiary
which were planned real time was not possible even virtually due to network issues in the field. The team had
to resort to deferred virtual assessment and facilitation, usually in the afternoons or evenings after the
workers were back to their stations.
With the raging pandemic that required the FHWs to put in additional effort for COVID-19 screening and
vaccination, there was a significant increase in their workload and lack of a regular availability of the
workforce for the study purpose.There was also an underlying concern amongst the FHWs that the app is
likely to increase their workload without an increase in their remuneration. This feeling came in due to
required partial duplication of effort since the existing manual workflows continued and the FHWs had to
make data entries into the app for beneficiaries that met the inclusion criteria only for the purpose of the
study. This led to a slow uptake by the workforce. The FHWs did not fully understand that this was in a
study-mode and that the workload would decrease after the study, either by replacing the manual registers
with the app, or by going back to status-quo, which is the reason why some of the FHWs were hesitant and
not fully forthcoming on their feedback during the FGDs. The critical support by project execution partners
Piramal (PSMRI) in the field under direct supervision by NITI Aayog was greatly helpful in execution of this
project by convincing, motivating and monitoring the FHW participating in the study.
287887/2021/DM&A
525 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
45
Nascent technology implementation in geographies with infrastructural challenges
The project was testing implementation of a customised technical solution in a nascent area in an aspirational
remote district of the country. In addition to the customisation and adaptation it required a local vehicle for
frontline access. In the absence of any individual electronic health records, it required a local digital vehicle to
capture the patient or beneficiary related data. The local server then had to connect with the customised
CDSS tool in an anonymised way. This system architecture created many challenges as it required a well
synchronised functioning. As the Amrit app on which the CDSS tool (Arezzo) had to ride was also new and
in its agile evolutionary phase, there were time lags in identifying problems and providing solutions.
This first prototype developed under the project lacks many enhanced features in its user interface as
CDSS had been riding on a new app environment. The prototype app also did not have an edit
capability or the ability to go back and forth between screens, which resulted in the need for restarting a
journey if a data entry error was made. This requirement was unanimously voiced by most participants during
the survey as well as the FGDs and is a critical requirement for future releases of the CDSS app.
In particular, the need for due lists and appointment-scheduling was a significant requirement stated by the
FHWs. However, there was also some hesitation in expressing this feedback clearly, since the addition of
such features would increase the transparency in the system leading to a higher level of accountability on the
part of ASHAs and ANMs. Get data functionality essential for continuum of care from ASHA to ANM to
MO could be made available towards the end of study with limited time for the HCWs to assess its
usefulness.
There was also a learning curve with the app. The first visit involves registration of the beneficiary and
takes time to enter demographic and social information before a CDSS journey can be initiated. Thus several
participants reported usability issues, in terms of time, however, it was also evident that during subsequent
periods the workflow was more efficient and progressively the data entry requirements reduced and the
CDSS aspects of the app became more prominent, the journey itself took much less time. Moreover, the
CDSS app ensures detailed history which wasn‘t elicited earlier and the FHWs weren‘t accustomed to this,
but over the study period the benefits of this initial history became apparent, and several users gave positive
feedback about its usefulness, despite some issues with usability in the short term.
Network connectivity was a significant bottleneck, and it was evident that the clear preference is for an
app that can work in the offline mode. Technologically, it is possible to create an asynchronous app that has a
device-level memory and that can further be synchronized once network is available, however this kind of
app architecture was not available for the study and connectivity did impinge on the usability of the app
significantly.
Hesitancy and unfamiliarity withlikely benefits with new technology among beneficiaries
There were quite a few challenges with the CDSS workflow when beneficiaries presented as a group,
like in the case of VHNDs. While the expectation of beneficiaries is to be done within 2 or 3 minutes, the app
would require significantly more time, sometimes up to 30 minutes, and the beneficiaries were not willing to
wait for so long, due to their domestic and work-related priorities. Further, without a prior experience or clear
understanding of the benefit that such systems will provide to the beneficiary there was also no clear
trade-off that was apparent to the beneficiaries who were part of the study. The pertinent issue that emerges
here is what incentives do beneficiaries have to participate in a health system that is enhanced with
knowledge-based technology and addressing this would be a major implementation challenge in the future.
This in many ways requires a cultural change in the community and their expectations from the healthcare
delivery system. This comes with experience of the benefits and its word-of-mouth propagation within the
community. The value of preventive and promotive care in the minds of most people, including
educated people, is far less than the curativeservices. This is essentially the area that requires a cultural
shift in the community as well as the workers.
Study design and plan limitations
A major limitation of the study was that it this initial POC prototype was restricted to 10 conditions in
maternal and child health. This meant it was not possible to track all potential comorbidities and disease
287887/2021/DM&A
526 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
46
clusters. Several beneficiaries did present with other common conditions or were of a different age group and
the study design did not accommodate all of those due to its scope and inclusion criteria. An initial study had
to avoid an uncontrolled scope that would have impinged on the primary outcome measures of usability,
usefulness and adherence to STGs. However, it would be ideal to overcome this limitation in future study
designs to assess the impact of a wide coverage of conditions like communicable and non-communicable
diseases. It may be useful to do a pre study survey in the facilities to be included and list out all possible
common conditions with which patients under consideration present and include most of those so that the
HCWs use the app for all or most and not selectively.
There was also a limitation in terms of the geographic coverage of the study sites. Only a limited
number of SCs, PHCs, CHCs and their constituent health workers were a part of this study. This impacted
the follow-up of patients and referrals at the CHC, limiting the process of tracking referred patients.
Future implementation studies which will require some further degree and detail of customisation (e.g. being
mindful of local language, cultureand practices) shall need to be planned in a more participative manner
rather than a top down approach. The team may need to shadow-learn working of the end users and take
specific inputs from them before and during the study to fine tune the final product as per their working and
job needs. The planning period has to be fairly long and widespread to capture a detailed picture. The current
study could have resulted in a better product if we could have done all the planned interactions which got
impacted due to COVID. Learning from the improvement philosophy of Japanese industry - ―Kaizen‖ may
be pertinent here. It has three important components called 3G; which involves having the managers go to
the actual floor (gemba); looking at the actual products involved (gembutsu) and gathering as many facts about
the situation as possible (genjitsu) to plan improvements.
It is also important to consider all the preparatory elements required for the study. The need for a stable
app, local language translations, technical integration to be completed prior to the commencement of the
study was a key learning lesson from this study. Further the study site selection, transition period for training,
absorption of technology, as precursors of the study would ensure a higher level of preparedness before
initiating the actual implementation process and ensuring that implementation challenges do not
obscure the actual benefits of the study. The preoccupation of the HCWs with their daily activities
interspersed with focused activities left them with limited time to focus on learning this new technology.
Familiarizing all HCWs with electronic data management of all beneficiaries would be a welcome first step for
sensitizing them to this concept. Use of CDSS could then follow.
Our study was designed as a qualitative feasibility study and did not directly address the impact of
CDSS on health outcomes. After having conclusively ascertained the feasibility of implementing CDSS in
public healthcare settings, through primary outcome measures of usability, usefulness, and adherence to
STGs, future research efforts into CDSS need to be directed towards impact assessment, aimed at the
direct effect of a CDSS intervention on health outcomes.Such kind of an impact assessment would
require a different study design and would need to be conducted over a longer duration since the effect of
CDSS on health outcomes is a gradual process and it takes time for such interventions to start becoming
visible in direct health outcome measures related to clinical effectiveness, morbidity and mortality. The
indicators used for the study were, firstly, on the perception of participants and, secondly, on adherence to
guidelines. An impact assessment study would entail verification of compliance on the ground, either through
secondary surveillance systems or direct supervisory monitoring, aspects that were out of the scope of this
study.
While a Cluster RCT with and without the intervention of CDSS could be a valid study design, this
methodology is fraught with the risk of uncontrollable confounding variables between two study sites
– in terms of locally relevant practices, language, dialect, infrastructure, healthcare administration and similar
factors.
A before and after study design could partially offset these confounders but this need be calibrated
with the right mix of study participants, clinical conditions, and an appropriate study duration.
Ideally a study design that has an initial ‗before‘ phase to evaluate current knowledge, attitude, practices and
health indicators needs to be sequenced with an ‗after‘ phase that has a considerable ‗run-in‖ phase
287887/2021/DM&A
527 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
47
toaccommodate the time it takes for any intervention to have an actionable impact on health outcome
measures. This duration needs to be ascertained by statistical extrapolation of existing health indicators and
ongoing health programs but is not likely to be lesser than 24-36 months.
Further the choice of primary outcome measures is critical. While the current study design included
quantitative STG adherence measures, these were based on compliance of FHWs to recommendations
contained within STGs. For a through impact assessment study the indicators would be more
proximal to the actual health status of beneficiaries and would be a combination of process and
outcome indicators. E.g. Anemia would include process indicators relating to the prescription of IFA, Anti-
helminthics, medication adherence at the community level, and outcome measures relating to the Hb trends
at the population level, and the impact of interventions on anemia over a period of time, since the
supplements would take at least 12 weeks to have a measurable impact on anemia at the population level as
well as in special groups like post-partum patients. Also, it would require that the outcome measures be cross-
functional since simple interventions in combating anemia would result in remarkable improvements in other
health indicators too.
The future studies certainly also need to take into account the technical issues related to functioning of the
app and the intelligent controller ecosystem (software, server, internet access, rapidity of response, etc.).
Expanding from learnings from the study, with inputs from the multi-disciplinary steering group, the way
ahead will need to use enhancements of the app as well as beyond it. The possible preparatory steps will need
development of, action oriented implementable guidelines to cover the whole gamut of primary health care
services and convert them to computer interpretable formso that the app replaces the paper registers in the
implementation study area completely. The tool should have capacity to be the first care step for patients to
get advice.
While that advancement happens, testing of the app with a large base of end-users may be useful to find,
execute and fine-tune some important technical solutions required to remove the stumbling blocks, e.g.:
a. Solutions to decrease latency and ensuring almost no denial of service from the server when multiple
users are using the app at the same time, due to which the requests may not be serviced by the app.
b. Solutions to decrease impact of lag due to slow internet connectivity in fringe areas by:
reworking out app with whole corpus of questions, prescriptions and other details being
locally present in the handset itself or have some other local caching mechanism available
over the site. It will also help to prevent fast drainage of battery owing to much network
traffic going back and forth.
using local edge client servers for making sure that static components of the content are
delivered from local edge servers, and dynamic components can be fetched from the cloud
periodically to ensure there is low latency in content fetching.
Possibly, an easy to use interface can also be developed to assist self-guidance of the patients, based on their
symptoms (like tele-medicine). Technology aided customized solutions have the potential for improving
access to appropriate care through intelligent decentralized guidance. The use of telemedicine has just started
during the current pandemic and definitely looks promising to continue as an additional tool for expanding
the reach of health care.
Beyond the context of the present study, we do realize that there are immense possibilities for use of
technological solutions to fill gaps and strengthen health care. Partners from technological, medical and field
implementation ecosystems within India should work together to create solutions using artificial intelligence
and machine learning. We need to test how far the conversational AI can be developed to improve the access
for those who find it difficult to read and respond.
Intelligent solutions could be devised to harness the information collected through community users to
identify early trends of a disease outbreak or to assess the health of the community from the nutritional data,
illness records, etc. The use of digital technology could not only help task sharing and shifting across the
various level of health functionaries but also between the man and machine, thus filling up some of the gaps
due to low numbers of trained doctors in the periphery.
287887/2021/DM&A
528 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
48
9. Conclusion
The present study conclusively demonstrated the feasibility to transform country guidelines and create
advance data analytics to develop acustomized clinical decision tool. It required the local experts to
harmonize and develop the guidelines to assist their metamorphosis to computer interpretable format. The
expertise of the creative partners at Elsevier ensured integration of “Arezzo®” - an active Declarative AI
clinical decision support system toan indigenous local app.Theadolopementprocess had utilized all the
elements viz. adoption, adaptation as well as development to create the final prototype.
We also established that the integrated CDSSis usable and useful to health workers in primary and
secondary healthcare settings and is preferred by over three fourths of the users for continued
adoption. Provision of Arezzo® customized for Indian guidelines-based recommendations empowered the
frontline worker as it acted as an able job aide with resultant improved recognition of high-risk cases in the
community. It showed added potential for efficient task share and task shift of care across the health care
functionaries from community to FRUs and vice versa. The study results are convincing enough to support
development of next version tools with further enhancements.
287887/2021/DM&A
529 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
49
10. References
1. World Health Organization. WHO Country Cooperation Strategy At a Glance: India. 2018.
2. World Health Organization. Maternal mortality - key facts. https://www.who.int/news-room/fact-
sheets/detail/maternal-mortality. Published 2018.
3. NITI Aayog. Maternal Mortality Ratio (MMR). https://niti.gov.in/content/maternal-mortality-ratio-mmr-
100000-live-births. Published 2018.
4. Lawn JE, Blencowe H, Kinney M V, Bianchi F, Graham WJ. Evidence to inform the future for maternal and
New-born health. Best Pract Res Clin Obstet Gynaecol. 2016;36:169-183.
5. Dhirar N, Dudeja S, Khandekar J, Bachani D. Childhood Morbidity and Mortality in India—Analysis of
National Family Health Survey 4 (NFHS-4) Findings. Indian Pediatr. 2018;55(4):335-338.
6. Soni M, Agrawal S, Soni P, Mehra H. Causes of Maternal Mortality: Our Scenario. J South Asian Fed Obstet
Gynaecol. 2013;5(3):96-98.
7. Organization WH. WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience. World Health
Organization; 2016.
8. Ministry of Health and Family Welfare G of I. Guidelines for Antenatal Care and Skilled Attendance at Birth by
ANMs, LHVs, SNs. https://www.nhp.gov.in/sites/default/files/anm_guidelines.pdf. Published 2010.
9. Ministry of Health and Family Welfare G of I. Janani Suraksha Yojna.
http://pib.nic.in/newsite/PrintRelease.aspx?relid=123992. Published 2015.
10. Ministry of Health and Family Welfare G of I. Janani Shishu Suraksha Karyakram : Saving Mothers and New-
borns.
11. Singh PK, Rai RK, Alagarajan M, Singh L. Determinants of maternity care services utilization among married
adolescents in rural India. PLoS One. 2012;7(2):e31666.
12. ICF II for PS (IIPS) and. National Family Health Survey (NFHS-4), 2015–16: India. 2017.
13. Mavalankar D, Vora KS. The Changing Role of Auxiliary Nurse Midwife (ANM) in India: Implications for Maternal and
Child Health (MCH). Indian Institute of Management; 2008.
14. Ministry of Health and Family Welfare G of I. Rural Health Statistics. https://nrhm-
mis.nic.in/Pages/RHS2018.aspx?RootFolder=%2FRURAL HEALTH STATISTICS%2F%28A%29 RHS -
2018andFolderCTID=0x01200057278FD1EC909F429B03E86C7A7C3F31andView=%7B09DDD7F4-80D0-
42E3-8969-2307C0D97DDB%7D. Published 2018.
15. World Bank Data. Rural population (% of total population).
https://data.worldbank.org/indicator/SP.RUR.TOTL.ZS?end=2018andlocations=INandstart=2018andview=b
ar. Published 2018.
16. Chavda P, Misra S. Evaluation of input and process components of quality of child health services provided at
24 × 7 primary health centers of a district in Central Gujarat. J Family Med Prim Care. 2015;4(3):352-8.
17. G Sugunadevi. Quality of antenatal care services at subcentres: an infrastructure, process and outcome evaluation
in a district in Tamil Nadu. International Journal Of Community Medicine And Public Health, 2017:4(11):4071-
4077.
18. Praveen D, Patel A, McMahon S, et al. A multifaceted strategy using mobile technology to assist rural primary
healthcare doctors and frontline health workers in cardiovascular disease risk management: protocol for the
SMARTHealth India cluster randomised controlled trial. Implement Sci. 2013;8(1):137.
19. Callaghan M, Ford N, Schneider H. A systematic review of task-shifting for HIV treatment and care in Africa.
Hum Resour Health. 2010;8(1):8.
20. Joshi R, Chow CK, Raju PK, et al. The rural Andhra Pradesh cardiovascular prevention study (RAPCAPS): a
cluster randomized trial. 2012.
287887/2021/DM&A
530 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
50
11. Annexures
Annexure 1: Arezzo® Validation
The regulatory authority in the UK (MHRA) has advised that Arezzo does not need to be registered as a
medical device currently. This is because: Arezzo provides explicit recommendations with clear rationales (i.e.
is not a black box); the clinical user can review all data that has been used by Arezzo; and the user can access
the primary source content at any time. External Legal Counsel has confirmed the same findings for the FDA
and the European regulatory authority.
The original research work for Arezzo® was carried out by Imperial Cancer Research Fund (ICRF) in the
UK. The work continued under Cancer Research UK, after the merger of ICRF. ICRF set up the Advanced
Computational Laboratory with the primary research objective of ensuring existing cancer treatments of
known efficacy are used to maximum benefit, whilst new treatments are being developed. The research and
early prototype work are set out in the book ‗Safe and Sound: Artificial Intelligence in Hazardous
Applications‘, written by Professor Fox and Dr Das (ISBN: 978-0-262-06211-4).
Following delivery of the first Arezzo prototype, a series of research studies were carried out. These studies
were the equivalent of phase 1/2 trials, seeking to understand what the potential benefits might be within the
research laboratory:
CAPSULE Study
• Research objective: determine potential impact of Arezzo on prescribing by GPs
• Research design: Decision support vs unaided choice; 42 UK GPs, who made choices from a simple
alphabetic list (control condition) or a computer-generated shortlist with a patient-specific rationale
(arguments for and against each option on the shortlist).
• Outcome/s: CAPSULE demonstrated a 70% increase in GP decisions that matched those of an expert
panel; a 50% increase in selection of a cheaper but equally effective drug, and decisions were made 15%
more quickly.
• Publication: ‗CAPSULE: Routine prescribing in general practice‘. Walton et al. British Medical Journal
1997, 315:791
CADMIUM Study
• Research objective: determine potential impact of Arezzo on detection of breast cancers during
mammography screening.
• Research design: Decision support vs unaided interpretation; 4 radiographers made decisions based on
mammogram images with and without CDS interpretation.
• Outcome/s: Radiographers trained to interpret mammograms performed better when using the advice
provided by CADMIUM: correct assessments of abnormalities were significantly increased and incorrect
judgements were significantly reduced.
• Publication: ‗CADMIUM: Detecting abnormalities in breast cancer screening‘. Medical Image Analysis,
Taylor et al 1999, 3 (4), 321-337.
RAGS Study
• Research objective: determine potential effect of Arezzo CDS on detection of familial cancer risk.
• Research design: Argumentation vs statistical vs paper and pencil; 36 GPs managed 18 simulated patients:
6 with a computerised decision support system (RAGS), 6 with an established pedigree drawing program
designed for clinical geneticists (Cyrillic), and 6 with pen and paper.
287887/2021/DM&A
531 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
51
• Outcome/s: RAGs resulted in significantly more appropriate management decisions than either Cyrillic
or pen and paper. RAGs also resulted in significantly more accurate pedigrees than the other methods. 33
doctors (92%) preferred using RAGs overall.
• Publication: ‗RAGS: Computer support for interpreting family histories. Emery et al British Medical
Journal, 2000, 321; 28-32.
LISA Study
• Research objective: improved accuracy of prescribing.
• Research design: Decision support vs unaided care; A web-based decision-support system was designed
to facilitate access to full blood count information across geographical locations and to assist with dosage
for maintenance chemotherapy. 36 clinicians with varied experience decided on oral chemotherapy doses
for 8 simulated cases: 4 with CDS; 4 without.
• Outcome/s: Significantly reduced the number of erroneous prescriptions (0/144 with CDS vs. 54/144
without; P <0.0001).
• Publication: ‗LISA: Chemotherapy decisions in acute lymphoblastic leukaemia‘. Bury et al British Journal
of Haematology, 2005, 129, 746-754.
• These studies suggested that Arezzo could provide benefit, which led to more rigorous randomized
controlled trials:
HAVANNA Study
• Research objective: reduction in HIV viral load.
• Research design: Cluster-randomised, multi-centre international controlled trial; HIV/AIDS patients
underwent viral genotyping; HIV specialists randomized to use CDS or clinical judgement to prescribe
anti-retroviral therapies.
• Outcome/s: 30% reduction in viral load in patients treated with Arezzo versus no CDS.
• Publication: C Tural et al. The Havana Trial. AIDS. 2002; 16:209-218.
FASTEST Study
• Research objective: reduction in all-cause mortality and vascular events; adherence to guidelines.
• Research design: Multi-centre, single-blind, cluster randomized, controlled trial (29 clinics) comparing
electronic decision support guided management with usual care for TIA/stroke.
• Outcome/s: More intervention patients received guideline-adherent care (131/172; 76.2%) than control
patients (49/119; 41.2%). 90-day stroke or TIA occurrence was lower in the intervention group (2.3%)
than the control group (8.5%). User feedback was positive.
• Publication: ‗Diagnosis and management of patients with acute stroke‘. Ranta et al Neurology 2015 14;
84(15), 1545-51.
Several non-regulatory validation studies have been carried out with the national implementations of
Arezzo®:
• Early Referrals Application (ERA) – implemented UK Department of Health guidelines for improving
early detection of suspected cancers. An independent evaluation of the solution, which was used by GPs
in Leicestershire, demonstrated a 30% reduction in inappropriate referrals and a significant improvement
in the quality of referral documentation, as determined by cancer specialists.
• NHS Direct Health and Symptom Checker – delivered patient-facing online health and symptom
checking CIGs via a website, syndication feeds, and mobile apps. In the first year the use of this service
resulted in 1.3M fewer visits to other healthcare services, including 0.7M visits to GPs, saving the NHS
287887/2021/DM&A
532 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
52
an annual total of £57M1. The solution also detected the second wave of the H1N1 pandemic in the UK
in late 2010. More than 65 million users utilized the solution over 5½ years, with no reported clinical
incidents.
• NHS24 telephone triage solution – is operating with Arezzo® live in Scotland since May 2016. The
solution delivers CDS to clinical and non-clinical call handlers. It has been used to support more than 2
million calls. The solution integrates more than 300 separate CIGs into a unified call journey, utilizing
Arezzo‘s innovative ability to merge CIGs. Call times have been reduced and a recent evaluation has
shown that significantly fewer patients require GP appointments or visits after using the service.
• Great Ormond Street Hospital for Sick Children (London, UK) – where Arezzo® is being used to
manage complex diagnostic endocrine and metabolic pathways, integrated with the electronic medical
record. The solution has improved the safety and efficacy of the pathways execution, as well as saved one
FTE junior doctor equivalent in resources.
In New Zealand where the solution is integrated with electronic medical records for more than 90% of family
physicians. It has been used for more than 8 years. New Zealand has one of the highest rates of childhood
asthma in the world, with 20% of children affected. In 2009 BPAC NZ established a Childhood Asthma
programme for GPs which included Arezzo® CDS to help record patient histories and give advice on
medication/hospitalisation. There was an average of 25.5 hospitalisations per 100 child-years prior to their
first assessment with the BPAC module, and an average of 12.1 hospitalisations per 100 child-years following
assessment, i.e. 2.1:1 hospitalisation before compared with after (95% CI = 1.2 to 3.7, p=0.01). At the same
time in Australia, Arezzo® CDS was used by the largest health insurance provide to support a national
workplace health solution. The solution resulted in a significant improvement in the quality of
recommendations, as well as the efficiency of the service.
287887/2021/DM&A
533 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
53
Annexure 2: Survey Questionnaires
For ASHAs:
General Details
Role ASHA Facility Name:
Age Group < 20yrs 20-45 yrs > 45yrs Location
Gender Male Female No. Years of Work
Experience
Language
Proficiency
Can Read-write and
understand well
Can read-write little
but understand well
Cannot read-write.
Understand well.
Cannot read-
write. Understand
little.
Cannot read-write.
Cannot
understand.
Hindi
English
Other:
Which language above:
# Questions Response
1.
How much time do you spend in advising women on
reproductive health, pregnancy and taking care of
young children including immunizations?
<5
min
5-10 min
1
0
-
1
5
m
in
15-30 min >30 mins
2. How many households to you visit weekly for above? <10 10-25 25-50 50-75 >100
3. Do you maintain records for above?
Yes
No
4.
Is recording manual or electronic?
Manual Electronic
5.
When a new treatment guideline is released, are you
given any training
Yes No
6.
If yes, what is the quality of training: Very Poor
1
Poor
2
Neutral
3
Good
4
Excellen
t
5
7. How many hours are allotted for training?
8.
When new guideline training is given, rate your
experiences with respect to:
Very Poor
1
Poor
2
Neutral
3
Good
4
Excellen
t
5
Retention of knowledge
Ease of recollecting
Ease of reading/referring to the manual
Ease of reaching out to doctor
9.
For making any recommendation, do you
contact/refer to
Doctor/Nurse Manual Both
10.
How often do you refer to the guideline after its
release/training?
Never
Only
once
2-3 times
As and
when
required
Always
till I am
well
verse
with it
11. Do you use any digital tool for daily routine? Yes No
12.
If yes, what is the name and its purpose?
(probe around: name, what is it for, integration with
daily routine, user experience, how does it help,
limitations)
13.
If there is a tool which will help you in making clinical
decisions/recommendations by personalized data
analysis based on guidelines, will it be helpful?
Yes No May Be
287887/2021/DM&A
534 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
54
General Details
14.
What could incent you to use a tool like this? (money,
time saving, help in becoming more
knowledgeable….)
15.
Which phone or tablet, if any do you use and since
how long?
(Type of handset: Feature/smartphones)
16.
How will you rate yourself in terms of:
Very Bad
1
Bad
2
Average
3
Good
4
Very
good
5
Reading from a mobile device/tablet
Typing into a mobile device/tablet
Navigating through multiple screens
17.
Any other suggestions and Feedback you would like
to provide
18.
What data do you need to provide on a regular basis
to the center? If this app can help collect data
electronically for automatic upload, will it be useful?
287887/2021/DM&A
535 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
55
For Doctors and Nurses
General Details
Role
(circle one)
Medical Officer / GNM / ANM Facility Name:
Age Group < 20yrs 20-60 yrs > 60yrs Location
Gender Male Female No. Years of Work
Experience
Language
Proficiency
Can Read-write and
understand well
Can read-write little but
understand well
Cannot read-write. Understand
well.
Cannot read-write.
Understand little.
Cannot read-write.
Cannot understand.
Hindi
English
Other:
Which language above:
# Questions Response
1.
How many patients on average do you see
daily?
<10 10-25 25-50 50-75 >100
2.
What is the average time spent in one
consultation (symptoms, diagnosis, taking medical
history, checking past medical reports, prescribing
medicine)
<5 min 5-10 min 10-15 min 15-30min >30 mins
3. Do you record patient’s past medical history?
Yes
No
4. Is recording manual or electronic? Manual Electronic
5.
How is the recording done?
(Probe around: registers, process, storage, retrieval, etc). If
electronic, tablet, mobile, computer).
6.
How is the quality of records in terms of:
Very Poor
1
Poor
2
Neutral
3
Good
4
Very Good
5
Completeness of the record
(All data fields are captured)
Correctness of the records
(All the information is correctly captured)
Legibility of the records
(Can read and understand the records to act)
7.
Continuing with Q7, what is the reason for poor
record quality (If Q7 score is 2 or less than that)
8.
Do you know what standard treatment
guidelines (STGs) are?
For Medical Officers:
9.
When a new treatment guideline (STG) is
released, how much time does it takes to study
and remember the guidelines?
10.
How often do you refer to the guideline after its
release?
Never
Only
once
2-3 times
As and
when
required
Always till I
am well
versed with it
11.
How do you monitor the STG adherence of your
staff (nurse/ANM/ASHA)?
(probe around – Who does it, frequency, is this data
documented, what action is taken compliance is
low/poor)
For Nurses:
287887/2021/DM&A
536 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
56
General Details
12.
When a new treatment guideline is released,
are you given any training
Yes No
13.
If yes, what is the quality of training: Very Poor
1
Poor
2
Neutral
3
Good
4
Excellent
5
14.
How many hours are allotted for training?
15.
When new guideline training is given, rate your
experiences with respect to:
Very Poor
1
Poor
2
Neutral
3
Good
4
Excellent
5
Retention of knowledge
Ease of recollecting
Ease of reading/referring to the manual
Ease of reaching out to doctor
16.
For making any clinical
decisions/recommendations do you
contact/refer to
Doctor Manual Both
17.
How often do you refer to the guideline after its
release/training? Never
Only
once
2-3 times
As and
when
required
Always till I
am well
verse with it
For everyone:
18.
Do you use any digital tool for daily routine?
Yes No
19.
If yes, are you comfortable using this tool?
Yes No
20.
If yes, which tool and for what purpose?
(probe around: name, what is it for, integration with daily
routine, user experience, how does it help, limitations)
21.
If there is a tool which will help you in making
clinical decisions/recommendations by
personalized data analysis based on guidelines,
will it be helpful?
Yes No May Be
22.
What benefits do you foresee with the above
tool in your daily routine?
23.
What according to you will promote its uptake
among the other doctors/nurses/ASHA workers?
24.
Which phone/tablet do you use and since
when?
25.
How will you rate yourself in terms of? Very Bad
1
Bad
2
Average
3
Good
4
Very good
5
Browsing the internet
Typing/texting on phone or tablet
26.
Any other suggestions and Feedback you would like to provide:
27.
What data do you need to provide on a regular
basis to the center? If this app can help collect
data electronically for automatic upload, will it be
useful?
287887/2021/DM&A
537 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
57
Final Scoreard:
Please rate the following about your experience with the clinical decision support tool:
Strongly
Disagree
1
Disagree
2
Neutral
3
Agree
4
Strongly
Agree
5
Overall, I was satisfied with the CDS
tool
Usability
1 It is easy to use
2 The information is organized and
displayed in a logical manner
3 Recording patient data and
navigating through the workflow is
seamless
4 I am able to access the particular
guidelines as and when I need
them.
5 I feel comfortable using this tool in
social/community setting
6 The tool integrates/fits easily into
my daily routine
7 I am able to accomplish my tasks
more easily by using this tool
Usefulness
1 My productivity has increased
2 The quality of my interactions has
reduced due to the time spent in
entering data and reading
instructions from the tool
3 The toolgives me more information
on what toadvise patients than I
possess
4 I am more confident when I talk to
patients about their conditions and
recommendations
5 I am able to accomplish my tasks
more quickly
6 This tool helps me remember all
diagnostic procedures to be
advised
7 This tool helps me make better
clinical decisions
8 This tool helps me administer the
right drug at the right dose
9 This tool enables me to determine
which patients should be referred
10 I often need to override the
suggestions made by the tool
11 I would like to continue using this
tool for my daily routine and work
Other Comments:
287887/2021/DM&A
538 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
58
Annexure 3: Baseline Survey Results
ASHAs
The average age of the 20 female ASHAs was 29.4 years and their average experience was 5.4 years.
All 20 (100%) of them could read, write, and understand Hindi well, but only 3 (15%) could read, write, and
understand English well. 13 (65%) could not read or write English, but understood a little, whereas 2 (10%)
could not read, write nor understand English. It was thus evident that Hindi was the appropriate medium of
instruction and not English.
40% of the ASHAs visited less than 10 households a week,
30% visited between 10-25 households and 30% visited
between 25-50 households.
The average time spent on advising women on reproductive
health, pregnancy and childcare including immunizations was
10-15 minutes.
All 20 (100%) of the ASHAs maintain manual records in the
form of registers that they have access to for tracking
individual or family health. The registers have separate
sections for maternal and child health, where they record
current symptoms as narrated by the beneficiaries.
All 20 (100%) of the ASHAs knew what Standard Treatment Guidelines (STGs) are. 7 (35%) of the ASHAs
were trained on STGs one year ago, 3 (15%) were trained six months ago, 9 (45%) were trained less than
three months ago and 1 (5%) was trained less than one month ago.
0
5
10
15
20
Can Read-write
and understand
well
Can read-write
and understand
a bit
Cannot read-
write.
Understand well
Cannot read-
write.
Understand
little.
Cannot read-
write. Cannot
understand
Language Proficiency
ASHA
HindiEnglish
40%
30%
30%
Weekly Household Visits
ASHA
less than 1010-2525-50
287887/2021/DM&A
539 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
59
There is a wide variation on the hours spent on training - while 50% of the ASHAs responded saying they
spent 3-5 hours, 40% spent 1-2 hours and 10% spent 2-3 hours.
However, majority of the ASHAs (80%) perceived this training on STGs to be excellent.
As for the level of confidence in the field after training – 55% of the ASHAs felt a little confident and 45%
felt very confident, and none of them reported having low or no confidence.
7
3
1
7
12
months
6 months
3 months
less than
3 months
Last time ASHAs were
trained on STGs
1 1 3 1 2
5
6
1
0
2
4
6
8
9 8 7 6 5
1 or 2
hardly any sometimes
Number of training sessions received
on STGs
40%
10%
50%
Hours Spent on Training by ASHAs
3-5 hours2-3 hours 1-2 hours
80%
10%
10%
Qualty of STG Training as per
ASHAs
ExcellentGood Average
45%
55%
ASHAs' level of confidence in the field
after training
Very ConfidentA little confident
287887/2021/DM&A
540 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
60
Post new guideline training all the 20 (100%) of the ASHAs responded that they refer to the STGs as and
when required. The ease of referring to paper-based manuals was perceived to be good by 55%, excellent by
30%, and average by 5%, while 10% were not sure of this.
Their overall perceived retention of knowledge was good (95%), but the ease of recollecting was divided
between neutral (50%) to good and excellent (25% each). The ease of reaching out to doctors was perceived
as good (100%).
All 20 (100%) of the ASHAs responded positively to receiving handholding on the field. In case they are not
clear about STGs, 80% of them ask questions about STG and implement guidelines based on their
understanding, 15% of them are only mobilizing patients to health centres. All 20 (100%) of the ASHAs
consult with either the Medical Officer, ANM or the Nurse for giving any care advice, treatment, or referral.
None of the ASHAs used any digital tools for their daily routine at the time the base-line survey was
conducted. 60% of the ASHAs responded ―don‘t know‖ when asked if it would be helpful to have a digital
tool that helps them make recommendations through personalized data analysis based on STGs. 30%
responded positively and 10% responded ―may be‖. Thus it was evident that the level of awareness to the
potential of digital health was low, however there were 6 ASHAs who responded positively prior to the Phase
2 kick-off and the hypothesis was that this number would increase post the phase 2 study completion.
0
5
10
15
20
25
Retention of
knowledge
East of recollectingEase of
reading/referring to
the manual
Ease of reaching out
to the doctor
ASHAs experience with new Guideline Training
Very PoorPoorNeutralGoodExcellent
80%
15%
5%
Mechanisms to clear doubts about STGs
Ask questions about STG and
implement based on
understanding
Only bring patients to health
centers
Blank
287887/2021/DM&A
541 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
61
The biggest perceived benefits to use such a digital tool were to save time and increase their knowledge, with
9 (45%) of the ASHAs responding positively to this. However, 11 (55%) of the ASHAs had perceived
difficulties using such a tool, including 3 who thought it was not possible to use this tool at the site.
Only 5 (25%) of the ASHAs used smart phones and none of them had used a tablet device. Majority of the
ASHAs rated themselves average or below average in terms of various functionalities of mobile devices or
tablets.
All 20 (100%) ASHAs responded positively when asked If an app can help them collect data electronically for
automatic upload.
Subjectively, the suggestions and feedback of ASHAs included:
“Consulting ANM or doctor is what I always do. I am ready to get trained and work on the tablet.”
“Sometimes it is a bit cumbersome to transfer the patient to the doctor as we have to wait for them and sometimes the patient
denies to go, it takes a lot of time to convince them.”’
“I love the nature of my work and feel excited and happy doing it.”
“I have never used tablets and never entered data electronically. So I am excited to look forward for the training on the tablet and
then implement it in the field.”
30%
10%
60%
If there is a tool which will help you in making
any recommendations by personalized data
analysis based on guidelines, will it be helpful?
YesMaybeDon't Know
0
5
10
15
20
Very Bad Bad Below
Average
Average Good Very Good
Mobile device proficiency self-rating by ASHAs
Reading from a mobile device/tabletTyping into a mobile device/tablet
Navigating through multiple screens
287887/2021/DM&A
542 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
62
“Consulting ANM or doctor is always satisfactory for us and I am very positive toward the role I have been given and I find no
difficulty in doing the same. I am ready to get trained and work on the tablet. I think this will take less time as against the
manual method we use.”
“My suggestion would be to provide features of camera, watsapp in the tablet so that if there is no network atleast we can click the
photo of the information of the patient and later feed the information into the tablet. Alsowatsapp would help in sending the
information to the higher authorities.”
ANMs
The average age of the 21 female ANMs was 30.3 years and their average experience was 3.5 years.
All 21 (100%) of them could read, write, and understand Hindi well; 17 (81%) could read, write, and
understand English well, and 4 (19%) could understand English well, but could read and write a little. It was
thus evident that English proficiency was much higher in ANMs as compared to ASHAs.
2 ANMSs (9.5%) saw between 10-25 patients daily in the preceding week, 14 ANMs (67%) saw between 25-
50 patients per day, whereas 3 (14%) saw between 50-75 patients per day and 2 (9.5%) saw more than 100
patients per day in the preceding week.
35% of these patient interactions took 10-15, minutes
40% took 15-30 minutes and 25% took more than 30
minutes. The average time spent on advising women on
reproductive health, pregnancy and child-care including
immunizations was 10-15 minutes.
While 20 out 21 ANMs maintain manual medical
records of beneficiaries in registers, one ANM maintains
records both manually and electronically on the phone.
ANMs maintain manual records in the form of two
registers - ANC and master registers – that are stored in
the almirah of the office and can be accessed any time
for tracking individual or family health.
All 21 (100%) of the ANMs knew what Standard
Treatment Guidelines (STGs) are. 7 (33%) of the ANMs
were trained on STGs one year ago, 4 (19%) were
trained six months ago, 7 (33%) were trained less than
three months ago and 1 (4%) did not remember.
0
10
20
30
Hindi
English
Language Proficiency
ANM
Can Read-write and understand wellCan Read-write little and understand well
0.0%
10.5%
73.7%
15.8%
Daily patient load in preceding
week for ANMs
less than 1010-2525-5050-75
287887/2021/DM&A
543 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
63
ANMs receive training manuals which they use as and when needed for self-study after the training sessions.
4-5 hours are allotted to re-training or post-training support. Majority of the ASHAs perceived this training
on STGs to be good (62%) or excellent (32%).
71% of the ANMs hardly refer to these guidelines once published and training concluded, 19% as and when
required, and only 10% refer to the guidelines always until they are well versed with it.
ANMs overall perceived retention of knowledge, ease of recollecting, ease of referring to the manual were all
good (100% each).
However, on the ease of reaching out to Doctors 15 (71%) were neutral and 5 (28%) rated this as good,
showing definite opportunities for improvement in access to Doctors for ANMs, especially given the fact that
71% of them reach out to Doctors for making any clinical decisions/recommendations.
0
1
2
3
4
5
6
7
1 year ago
6 mos ago
3 mos ago
less than 3 mos ago
Not trained
7
4
2
7
0
Last time ANMs were trained on STGs
0
5
10
15
20
25
ANMs experience with training
on newly published guidelines
Retention of knowledge
East of recollecting
Ease of reading/referring to the manual
Ease of reaching out to the doctor
19%
10%
71%
Frequency of referring to the
guidelines after release/training?
As and when required
Allways till I am well versed with it
hardly anytime
287887/2021/DM&A
544 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
64
Only one ANM reported using a digital tool, but all 21 of
them are comfortable using digital tools. 62% of the
ANMs responded ―may be‖ when asked if it would be
helpful to have a digital tool that helps them make
recommendations through personalized data analysis based
on STGs. 5% responded positively and 10% responded
―may be‖.
16 (76%) of the ANMs used smart phones and 5 (24%)
used feature ‗dumb‘ phones. None of them used a tablet
device. 14 (67%) of the ANMs rated themselves good in
terms of various functionalities of mobile devices or
tablets, whereas 4 (19%) rated themselves are very bad.
Subjectively, the suggestions and feedback of ANMs included:
“Training sessions are very much helpful for us. When we find difficulty, we ask the trainers. Also, we look at the guidelines and
material given to us during training.
“There is a weekly AAA meeting i.e. ASHA, Anganwadi and ANM from 9 to 4 pm. It is scheduled for every Thursday and
all the weekly records (pregnant women and 0-5 years children) of ANM are tallied with the ASHA's records. All the ASHA
diaries are checked. The list of children which are due for vaccination are also discussed.”
“I would love to work on the tablet if training is given. I find training very useful for execution of our work.”
“In case of vaccination day, we are very busy. There is no private space to take the fundal height.”
Medical Officers
There were six male MOs and 3 female MOs who participated. All 9 of them could read, write, and
understand English and Hindi well.
Four MO‘s saw between 50-75
patients daily in the preceding
week, 4 MOs saw more than 100
patients per day, whereas 1 MO
saw between 25-50 patients per
day in the preceding week.6 MO‘s
took 10-15 minutes per patient
interaction whereas 3 MO‘s took
15-30 minutes.
All 9 MO‘s maintain manual medical records of beneficiariesincluding current symptoms as narrated and
additional screening questions on common and relevant illnesses. None of them used any digital tools for
daily routine work, however all of them were comfortable using digital tools.
(What benefits do you foresee with the above tool in your daily routine?)
All 9 MO‘s (100%) thought it would be helpful if there is a tool which will help them in making any
recommendations through personalized data analysis based on guidelines.
However, 8 of the MO‘s thought the use of the tablet-based digital health tool was questionable and 1
responded ‗don‘t know‘ to the question about promoting the uptake of such a tool amongst
doctors/nurses/ASHA workers.
1
4
4
Daily patient load in
preceding week for MOs
25-5050-75more than 100
6
3
Average time required per patient
interaction
10-15min15-30min
287887/2021/DM&A
545 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
65
All 9 MO‘s knew what Standard Treatment Guidelines (STGs) are and take 15-30 minutes to study and
remember it and then refer to it as and when required.
6 of the MOs rated themselves good in terms of various functionalities of mobile devices or tablets, whereas
2 of them rated themselves as very bad.
The MO‘s overall feedback was evident from their responses for suggestions - 4 were not interested, 3 were
neutral, 1 could not say if it would work and 1 was interested. The feedback from MO‘s also was that their
schedules were very busy especially with COVID-19 duties.
0
2
4
6
8
1 6 0 0 0 1 1
Mobile device proficiency self-rating by MO's
Typing into a mobile device/tablet
Navigating through multiple screens
Any other suggestions and Feedback you would like to provide
287887/2021/DM&A
546 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
66
Annexure 4: Phase 1 testing case mix
Maternal Cases (Retrospective from record review during Lockdown Period) n=82
No. of Journeys 357 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases = 82
ANC-without any complications 32
ANC with 1 or more
complications
2. Anemia in Pregnancy
3. Preeclampsia/Hypertens
ion Eclampsia
4. Gestational diabetes
34
5. 10 *
6. 16 **
7. 11
Post-natal 16
* These 10 women with anemia included 7 with only anemia and 3 women with anemia and preeclampsia
** These 16 women with preeclampsia included 13 women with only preeclampsia and 3 women with
preeclampsia and anemia.
Paediatric Cases (Retrospective from record review during Lockdown Period) n=73
No. of Journeys 354 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases
(n=73)
Feeding and Immunization alone
5 *
Pneumonia
Pneumonia alone
With feeding and immunization
7
5
2
Diarrhoea
Diarrhoea alone
Diarrhoea with Sick Young
Infant
With sick Child
With pneumonia
With feeding and immunization
32 **
18
5
3
2
4
Care of the Sick child
Care of the sick child alone
With feeding and immunization
18
9
9
Sick Young Infant 11
* The actual cases assessed for this guideline were 20 (5 normal, 9 sick child, 2 with pneumonia, 4 with
diarrhoea)
287887/2021/DM&A
547 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
67
Maternal Cases done (Prospective): from OPD (English version) (n=71)
No. of Journeys 342 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases (n=71)
ANC- without any complications 21
ANC with 1 or more
complications
Anemia in pregnancy
Hypertension/Eclampsia
Gestational diabetes mellitus
39
19 *
12**
12***
Post-natal 11
* These 19 include 16 with only anemia and 3 with anemia and GDM
** These 12 include 11 with only Preeclampsia and 1 with preeclampsia and GDM
*** These 12 include 8 with only GDM, 3 with GDM and anemia and 1 with GDM and preeclampsia.
Paediatric cases done (Prospective): from OPD (English version) n=102
No. of Journeys 504Continuum of Care Journeys completed for ASHA ANM Medical Officer
Cases
(n=102)
Feeding and Immunization 23*
Pneumonia
Pneumonia alone
With feeding and immunization
7
5
2
Diarrhoea
Diarrhoea alone
Diarrhoea w Sick Young Infant
With Sick child
With pneumonia
With feeding and immunization
32 **
12
1
3
3
13
Care of the sick child
Care of the sick child alone
With feeding and immunization
15
12
3
Sick Young Infant 14
New-born Care 11
*The actual cases assessed for this guideline were 41 (23 normal, 3 sick child 13 with diarrhoea and 2 with
pneumonia)
287887/2021/DM&A
548 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
68
Maternal cases done (Prospective): from OPD (English and Hindi version) n= 55
No. of Journeys = 264 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases
(n=55)
ANC-without complications
10
ANC with 1 or more complications
Anemia in Pregnancy
Preeclampsia/Hypertension/Eclampsia
Gestational diabetes mellitus
32
13*
10**
10
Post-natal
13
*These 13 include 12 cases of only anemia and 1 with anemia and preeclampsia
** These 10 include 9 cases of only preeclampsia and 1 with preeclampsia and anemia
PaediatricCases done (Prospective): from OPD (English and Hindi version) (n=56)
No. of Journeys 351(continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases (n=56)
Feeding and Immunization 5*
Pneumonia
Pneumonia alone
With feeding and immunization
10**
1
9
Diarrhoea
Diarrhoea alone
Diarrhoea with SYI
With Sick child
With Pneumonia
With feeding and immunization
12
1
4
0
0
7
Care of the sick child
Care of the sick child alone
With feeding and immunization
11
1
10
Sick Young Infant 7
New-born Care 11
* The actual cases assessed for feeding and immunization were 31(5 normal, 10 sick child, 9 with
pneumonia,7 with diarrhoea)
287887/2021/DM&A
549 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
69
The breakdown of the number of journeys is detailed below:
Maternal
Total number of Cases: 208
Total number of Journeys: 963
Journeys Breakdown: ASHA ANM MO
ANC-w/o complications 63 63 63
ANC w complications 105 105 105
ANC in Pregnancy 42 42 42
ANC-PE/HTN/EC 38 38 38
ANC-GDM 33 33 33
Post-natal 40 40 40
Pediatric
Total number of Cases: 231
Total number of Journeys: 2172
Journeys Breakdown: ASHA ANM MO
Feeding 92 92 92
Immunization 92 92 92
Pneumonia 29 29 29
Diarrhoea 76 76 76
Care of the sick child 50 50 50
Sick Young Infant 42 42 42
New born child 22 22 22
287887/2021/DM&A
550 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
70
Annexure 5: Interim Survey Results
A qualitative survey was executed in early March.
Usability
The overall positive response rate where respondents agreed or strongly agreed on the usability of CDSS-
integrated mobile app was 77.17%.
Usefulness
The overall positive response where respondence agreed or strongly agreed to useful features or disagreed or
strongly disagreed on reverse questions on the usability of CDSS-integrated mobile app was 72.95%.
2
3
1
2
0
2
6
4
2
4
5
6
4
8
3
6
4
1
19
20
26
23
22
23
15
11
14
11
15
14
05101520253035404550
6. I can accomplish my tasks more easily (मै …
5. It integrates/fits easily into my daily routine
(यह मेरी ददनचयाा मे आसानी में उपयुक्त …
4. I feel comfortable using it in
social/community setting (मै रोगी/ ऱाभाथी …
3. Recording patient data and navigating
through the workflow is seamless …
2. The necessary information and summary is
organized and displayed in a logical manner …
1. It is easy to use this mobile application (इस …
Usability N=46
Strongly DIsagree ??????Disagree ??????Neutral ??????Agree ??????Strongly Agree ??????
287887/2021/DM&A
551 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
71
82.61% of the respondents would like to continue using the app in their daily routine and the overall
satisfaction from the CDSS-integrated mobile app was 84.7%.
287887/2021/DM&A
552 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
72
Annexure 6: STG Adherence Secondary Data Analysis on limited journeys
Paediatric cases
The risk status of infants 0-2 months is based on birth weight, period of gestation, and their ability to
feed on day 1, and enables recommendation of an increased number of new-born home visits, and
additional care advice.
Assessment of the first feed was documented for 9 neonates. 100% reported successful breastfeeding
within 1 hour of birth.
First feed: (n = 9 neonates)
within 1 hour of birth 9
breastfed 9
suckled effectively 9
Gestational age at birth was documented for 75 infants, with 2 identified as high risk due to pre-term
birth:
Birth weight was recorded for 53% of infants aged 0-2 months:
One infant was identified as high risk, due to a birth weight < 2500g:
Across all age groups, a current weight measurement was documented at 11% of encounters:
287887/2021/DM&A
553 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
73
Weight-for-age status was automatically derived for children aged 2-60 months, according to WHO Child
Growth Standards. 16% of the children were identified as severely underweight, and 17% as moderately
underweight, for their age:
Symptom assessment for fever, pneumonia and diarrhoea-related diseases, and possible serious bacterial
infection in neonates, formed part of each encounter.
Temperature was measured at 15% of encounters:
If a temperature measurement was not documented, the FHWs were prompted to assess skin
temperature to touch. 19 cases of fever were documented: 4 based on a temperature measurement >=
37.5C; 15 based on assessment of the skin to touch.
5 cases of diarrhoea were documented across all encounters, with 1 recorded as persisting for 14 days or
more:
Diarrhoea (n = 1455)
acute (< 14 days) 4
persistent (>= 14 days) 1
with blood in stool 0
none 1450
None of the cases recorded significant signs of dehydration. A prescription of zinc was documented in 3
of the cases.
Symptoms of acute respiratory infection were assessed at encounters with children 2-60 months. 20 cases
of cough, breathing difficulty, or both were documented:
Symptom (n = 1320)
cough 19
breathing difficulty 3
none of the above 1300
287887/2021/DM&A
554 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
74
Further assessment identified symptoms consistent with fast-breathing pneumonia in 4 cases, and severe
pneumonia in 2 cases:
Encounters with infants 0-2 months included documentation of signs consistent with possible bacterial
infection. The frequency of positive findings is shown in the below table:
Sign Positive n
bulging fontanelle 12 162
cry weak or stopped 12 140
severe chest indrawing 1 155
frequent vomiting 1 140
large boil 1 148
crying incessant 1 140
redness of umbilicus 1 148
pus around umbilicus 0 148
jaundice 0 148
skin pustules 0 148
moving less or not at all 0 147
ear draining pus 0 147
abdomen distended 0 147
The outcome, or endpoint, of an encounter falls under three categories: urgent referral (or inpatient treatment
if appropriate) for severe symptoms, non-urgent referral for further investigation, and general home care
advice.
Urgent referral was recommended for 3% of cases, and non-urgent referral in 34% of cases:
287887/2021/DM&A
555 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
75
A breakdown of the reasons for urgent referral is shown below. ―Possible sepsis‖ refers to the presence of 1
or more severe signs, or 2 or more mild signs, of infection in an infant aged 0-2 months.
Of the cases where urgent referral was recommended, 51% documented that the child was unable to attend.
Reasons reported included unavailability of transport (3 cases), family reason (6 cases), and other individual
reasons (10 cases).
1
1
1
1
1
1
1
1
2
2
2
3
3
7
23
0 5 10 15 20 25
moderately low weight for age
persistent diarrhoea
poor weight gain
possible serious bacterial infection
signs of very severe febrile disease
unable to drink or feed
vomiting everything
weight less than 1800g
persistent fever
signs of pneumonia
signs of severe pneumonia
fever with respiratory symptoms
low body temperature
severe malnutrition
possible sepsis
Reasons for urgent referral
287887/2021/DM&A
556 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
76
Maternal cases (antenatal)
LMP was recorded in all but 10 antenatal cases:
For the cases where LMP was documented, 90% had a date recorded. 10% of women did not know their
LMP date:
All cases where the LMP was documented also had the EDD documented, including for women who did not
know the LMP date.
Yes, 18
No, 19
Availability of urgent referral (n=37)
287887/2021/DM&A
557 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
77
Women who did know the LMP date were asked for their best guess of the period of gestation in months. 11
women had an ultrasound to confirm the period of gestation, which was calculated as: 17 weeks (2 cases); 30
weeks (3 cases); 35 weeks (1 case); 38 weeks (1 case); and 40 weeks (1 case).
Symptoms were assessed by ASHAs and ANMs. The number of times that questions were asked varied
depending on the period of gestation, etc. This reflected the ability of the AI solution to understand the
context in which the questions were being asked, including the role of the User:
The range of answers was extended for some questions. Rather than just ‗yes‘/‘no‘ responses, the concepts of
‗bothersome‘ and ‗not bothersome‘ were used to distinguish the impact of some symptoms:
Both women who reported bothersome pain had continuous symptoms: 1 had mild pain; the other woman
had severe pain. The three women who had bothersome vomiting did not have persistent symptoms.
The frequency of positive findings for the other symptoms is shown in the following table:
287887/2021/DM&A
558 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
78
Symptom Present
Jaundice (symptom) 0
Oedema 1 (generalised)
Rash/itching 0
Epigastric pain 0
Excessive thirst* 12
Increased urinary frequency 7
Dysuria 2
Polyuria 0
Reduced urine output 0
Vaginal bleeding 2
White vaginal discharge 0
Blurring of vision 0
Watery fluid PV 2
*None of the women who reported excessive thirst had glycosuria
Symptoms of anemia were assessed with questions about dizziness, palpitations, and fatigue:
Bothersome symptoms No. Hb levels
Palpitations 1 Hb: 10.1
Fatigue 3 Hb: 7.8; Other 2 cases were >10
Dizziness 4 Hb: 9; 11; 11; 12
Breathless at rest* 2 Hb: 9; 11
*No longer asked as symptom of anemia but included for comparison
The question on palpitations includes the option to rate the symptom as ‗not bothersome‘. The following
chart shows the hemoglobin levels distribution for 16 women:
The question on easily fatigued includes the option to rate the symptom as ‗not bothersome‘. The following
chart shows the hemoglobin levels distribution for 42 women:
287887/2021/DM&A
559 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
79
Previous obstetric history was recorded for 123 women. The following table shows the number of previous
pregnancies:
No. previous
pregnancies (n=123 women)
0 4
1 42
2 32
3 18
4 12
5 10
6 3
7 1
8 0
9 1
The number of previous children is recorded in this table:
No. previous children (n=123 women)
0 8
1 40
2 32
3 19
4 13
5 8
6 3
Additional details of obstetric history are shown in the next table:
Obstetric history 'Yes'
Abortion 78
Previous Caesarean section 2
Previous complications 2
Previous neonatal death 1
Previous stillbirths 2
287887/2021/DM&A
560 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
80
Past medical history was recorded for 180 women:
Past medical history „Yes' No. asked
TB 1 180
Diabetes 2 180
Heart disease 1 180
Tobacco use and alcohol consumption was recorded for 180 women:
Social history „Yes'
Alcohol 1
Tobacco 16
Smoking tobacco 6
Smokeless tobacco 16
Family history was documented as completed for 180 women. There were no cases of hypertension, diabetes,
multiple pregnancies, TB, thalassemia, or mental retardation recorded in any families.
Obstetric risk assessment was performed in 213 cases. This is lower than the total number of antenatal cases
but reflects the fact that ASHAs do not perform risk assessments.
218 antenatal visits were performed by ANMs or MOs, which means that obstetric risk assessment was
recorded for 98% of the women seen by someone who is able to perform a risk assessment.
The proportion of high versus low obstetric risk cases is illustrated in the next chart, with the majority of
women (60%) having a high obstetric risk.
The details of the positive risk factors are set out in the following table, with several women having more
than one risk factor (209 risk factors distributed across 128 women):
High risk factors
No.
recorded
Anemia - mild 47
Anemia - moderate 40
Small-for-dates (good dating) 25
Smokeless tobacco usage 16
Previous neonatal loss 9
Foetal heart sounds absent 8
Large-for-dates (good dating) 8
Fundal height not increased 7
Smokers 6
287887/2021/DM&A
561 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
81
Fever 4
Easy fatigability 3
Previous difficult labour 3
Tachycardia 3
Abdominal pain 12 EDD 2
Breathless at rest 2
Headache 2
Pallor 2
Past history - diabetes 2
Past history - hypertension 2
Premature birth 2
Previous eclampsia 2
Small-for-dates (uncertain dating) 2
Alcohol consumption 1
Foetal heart sounds reduced 1
Generalised oedema 1
Other illness 1
Palpitations 1
Past history - TB 1
Past history - Heart disease 1
Previous delivery with congenital
anomaly 1
Previous pre-eclampsia 1
Rh negative 1
Vaginal bleeding <20 weeks 1
Vaginal bleeding >=20 weeks 1
219 women had body weight recorded in a visit, though 2 values were excluded as possible data-entry errors
(11 kg and 121 kg respectively). The following chart displays the distribution of maternal body weight, with a
median body weight of 48 kg:
287887/2021/DM&A
562 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
82
Maternal blood pressure was recorded in 221 visits. The distribution of systolic blood pressure readings is
displayed in the next chart. There is a bimodal distribution, suggesting an effect related to the measuring
device and/or the blood pressure measurement process:
All women who had elevated systolic blood pressure readings had repeat values taken during the visit. Two
women had persistent mild elevated blood pressure and were recommended for referral (see detailed Referral
Reasons section below).A similar bimodal distribution was seen for diastolic blood pressure readings.
Information was available on hemoglobin testing for 216 women. 197 women had hemoglobin
measurements. The distribution of maternal hemoglobin levels is shown below:
Blood group and Rhesus factor status was only measured in 3 women. In 174 visits, it was recorded that
blood group was not available.
VDRL testing was carried out in 174 women and recorded as ‗Not done‘ for 38 women. All tests that were
performed had negative results.
HIV testing was carried out in 196 women and recorded as ‗Not done‘ for 16. All tests that were performed
returned negative results.
Urine glucose was recorded as ‗ordered‘ in 179 women, with all results returning as negative for glycosuria. 34
women were documented as having the test ‗not ordered‘:
287887/2021/DM&A
563 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
83
Urine protein was recorded as ‗ordered‘ in 183 women, with all results returning as negative for proteinuria.
30 women were documented as having the test ‗not ordered‘.
Urine testing for glucose and albumin is carried out by ANMs and MOs. The tests are carried out in the
majority of visits:
Abdominal examination was carried out in 226 visits, with findings other than fundal height being recorded in
22 women. Nine women had an abdominal scar; 8 vertical and 2 ‗other‘. Foetal lie and presentation, as well as
foetal movements were also examined, depending on the period of gestation:
Foetal lie and presentation
Cephalic 67
Breech 2
Cannot tell 4
Foetal movements
Normal 110
Reduced 2
IFA therapy was reviewed in 225 women, with 8 women recorded as not taking IFA:
287887/2021/DM&A
564 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
84
Only one woman was recorded as having received parenteral iron therapy.
Tetanus toxoid vaccination status was reviewed in 187 women:
The AI solution calculated the scheduled date for a first tetanus toxoid dose in 43 women, for a second dose
in 3 women, and a booster date for 1 woman.
Deworming in the past 6 months was recorded as ‗done‘ for 123 of 201 women:not having treatment in the
second and third trimesters:
Deworming treatment was analyzed separately, with about 2/3rds of women having treatment in the first
trimester and 37% of women
287887/2021/DM&A
565 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
85
Counselling was recorded as being provided to 313 beneficiaries across the following topics. Note that
nutrition advice was provided on more than one occasion to some women:
Maternal cases (postnatal)
There were 44 women who were reviewed after delivery in February 2021. The demographic details have
been presented in a previous section of this document.
The following table shows the range of symptoms and other issues that were reviewed during the visits:
Postnatal review topics „Yes‟ „No‟
Regular diet started 18 11
Breast feeding baby 40 1
Lower abdominal pain 3 54
Breast feeding - insufficient milk 3 53
Breasts sore 1 54
Perineal pain 1 56
Breast lump 0 42
Breast tenderness 0 42
Breasts engorged 0 56
Breathlessness 0 57
Chills 0 56
Constipation 0 56
Contraception 0 24
Convulsions 0 42
Excessive bleeding at time of delivery 0 42
Dizziness 0 57
Epigastric pain 0 43
Easy fatiguability 0 41
Fever 0 56
Severe headache 0 43
Leg pain 0 55
Lochia abnormal 0 55
287887/2021/DM&A
566 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
86
Lochia foul-smelling 0 57
Mood changes 0 51
Nipples crusted or sore 0 57
Nipples inverted 0 55
Oedema 0 41
Pallor 0 41
Perineal swelling 0 57
Perineal tear 0 56
Urination burning 0 57
Urinary frequency increased 0 57
Urinary dribbling or retention 0 57
Uterus soft and tender 0 41
Vaginal bleeding 0 56
Blurring of vision 0 43
Body temperature of the mother was recorded during 52 visits. The following chart shows the distribution of
temperature in degrees Fahrenheit:
Blood pressure was recorded on 40 visits. The distribution of systolic and diastolic blood pressure values are
shown below:
Seven women had hemoglobin measured:
Hemoglobin measured
Yes 7*
No 10
Not known 27
*All 7 had mild anemia.
287887/2021/DM&A
567 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
87
Anemia treatment was recorded for 11 women; 31 women were noted as not receiving anemia treatment.
Calcium and vitamin D3 tablet usage was recorded in 57 visits:
Calcium Vit D3 tablets
One tab daily 35
Two tabs daily 21
No 1
The following advice was provided during 52 visits (91% of all visits):
Breastfeeding advice
Advice on calcium supplements
Family planning
IFA advice
Post-partum hygiene
Post-partum nutrition
Post-partum rest
Post-partum sexual activity
Family planning counselled was provided to 25 women. The number of women who agreed upon a family
planning method is shown in the following table:
Family planning method agreed upon
Yes 21
No 2*
Will decide later 2*
*No follow-up information is available for these women at the time of analysis
The family planning methods that the women agreed upon are listed in the next table:
Family planning methods
Centchroman* 9
Satisfied 6
Not satisfied 3
Condoms 12
Satisfied 12
COCP 0
ECP 0
IUCD 0
MPA 0
POP 0
*Centchroman is a family planning option developed in India
Referrals to a medical officer were recommended for the following reasons:
Recommended referrals to MO
Anemia 1
Suspected infection 5
Post-partum pain 3
___________________________________________________________________
287887/2021/DM&A
568 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
88
287887/2021/DM&A
569
A feasibility study to assess the usability,
usefulness, and adherence to Standard Treatment
Guidelines in Indian healthcare settings with the
use of Elsevier’s “Arezzo®” - a Declarative
Artificial Intelligence based Clinical Decision
Support System and pathway technology
Healthcare Artificial Intelligence Catalyst (HAIC) Pilot Study
Sponsored by NITI Aayog (GOI) & UK-DIT
Principal Investigator Team
Lady Hardinge Medical College and
All India Institute of Medical Sciences,
New Delhi
HAIC Pilot
Project Report
JULY 2021
287887/2021/DM&A
481 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
1
A feasibility study to assess the usability,
usefulness, and adherence to Standard
Treatment Guidelines in Indian healthcare
settings with the use of Elsevier’s “Arezzo®” -
a Declarative Artificial Intelligence based
Clinical Decision Support System and pathway
technology
Healthcare Artificial Intelligence Catalyst (HAIC)
Pilot Study
Project Report
July 2021
Sponsored by
NITI Aayog (GOI) & UK-DIT
Submitted by
Principal Investigator Team
Lady Hardinge Medical College and
All India Institute of Medical Sciences,
New Delhi, India.
287887/2021/DM&A
482 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
2
Table of Contents
List of Tables ............................................................................................................................................................ 4
List of Figures .......................................................................................................................................................... 4
List of Abbreviations .............................................................................................................................................. 5
Acknowledgements ...................................................................................................................................................... 8
1. Key Stakeholders and Project Team ................................................................................................................ 9
2. Introduction ....................................................................................................................................................... 11
2.1. Background ................................................................................................................................................... 11
2.2. Arezzo® ......................................................................................................................................................... 12
2.3. Rationale ........................................................................................................................................................ 13
3. Study Aim and Objectives ............................................................................................................................... 13
4. Hypotheses ......................................................................................................................................................... 14
5. Methodology ...................................................................................................................................................... 14
5.1. Study Phases .................................................................................................................................................. 14
5.2. Target Health Workers ................................................................................................................................ 15
5.3. Constitution of Expert Reference Group ................................................................................................ 15
5.4. Preparatory Field Visit ................................................................................................................................. 15
5.5. Infrastructure and Resources...................................................................................................................... 15
5.6. Selection and transformation of Guidelines ............................................................................................ 15
5.7. Detailed Methodology ................................................................................................................................. 16
5.7.1. Phase I ................................................................................................................................................... 16
5.7.2. Phase II ................................................................................................................................................. 18
5.8. Outcome Measures ...................................................................................................................................... 21
5.8.1. STG Adherence Indicators ................................................................................................................ 22
6. Results ................................................................................................................................................................. 24
6.1.1. Base-line Survey ................................................................................................................................... 24
6.1.2. Phase 1 Results .................................................................................................................................... 24
6.1.3. Phase 2 Results .................................................................................................................................... 26
6.1.3.1. Quantitative Scorecard Surveys Analysis .................................................................................................. 26
6.1.3.2. STG Adherence ................................................................................................................................... 32
6.1.3.3. Qualitative data: Focused Group Discussions ............................................................................... 35
7. Discussion .......................................................................................................................................................... 41
8. Lessons learnt and lateral insights for future studies .................................................................................. 43
9. Conclusion ......................................................................................................................................................... 48
10. References .......................................................................................................................................................... 49
11. Annexures .......................................................................................................................................................... 50
Annexure 1: Arezzo® Validation ....................................................................................................................... 50
287887/2021/DM&A
483 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
3
Annexure 2: Survey Questionnaires ................................................................................................................... 53
Annexure 3: Baseline Survey Results .................................................................................................................. 58
Annexure 4: Phase 1 testing case mix ................................................................................................................ 66
Annexure 5: Interim Survey Results ................................................................................................................... 70
Annexure 6: STG Adherence Secondary Data Analysis on limited journeys .............................................. 72
287887/2021/DM&A
484 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
4
List of Tables
Table 1 Target Health Workers
Table 2 Guidelines selected for the HAIC Pilot
Table 3 HAIC Study Participants
Table 4 HAIC Study Outcome Measures
Table 5 STG Adherence Indicators
Table 6 Phase 1 Reliability Testing Journeys
Table 7 STG Adherence Indicators for Maternal Health
Table 8 STG Adherence Indicators for Child Health
Table 9 Key Stakeholders of the HAIC Study
List of Figures
Figure 1 HAIC Study Gantt Chart
Figure 2 HAIC Study Flow-Chart
Figure 3 Baseline Survey Results
Figure 4 Phase 2 Qualitative Survey Results Summary
Figure 5 Usability Survey Results
Figure 6 Usability Survey Stacked % Results
Figure 7 Usefulness Survey Results
Figure 8 Usefulness Survey Stacked % Results
Figure 9 Overall Satisfaction Survey Results
Figure 10 STG Adherence Trend (Oct-2020 to April 2021)
287887/2021/DM&A
485 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
5
List of Abbreviations
AIDS Acquired immunodeficiency syndrome
AIIMS All India Institute of Medical Sciences
AMRIT Accessible Medical Records via Integrated Technologies
ANC Antenatal Care
ANM Auxiliary Nursing Midwifery
ANOVA Analysis of Variance
API Application Programming Interface
ASHA Accredited Social Health Activist
BP Blood Pressure
CDS Clinical Decision Support
CDSS Clinical Decision Support System
CHC Community Healthcare Centre
CI Confidence Interval
CIG Computer Interpretable Guideline
COVID-19 Novel Coronavirus disease (2019)
CVD Cardiovascular Disease
DBP Diastolic Blood Pressure
DMPA Depot Medroxyprogesterone Acetate
EDD Expected Date of Delivery
EHR Electronic Health Record
ERG Expert Reference Group
ETAT Emergency Triage and Treatment
FBNC Facility Based New-born Care
FGD Focus Group Discussion
FHW Frontline Health Workers
GDM Gestational Diabetes Mellitus
GOI Government of India
HAIC Healthcare Artificial Intelligence Catalyst
Hb Hemoglobin
HBNC Home Based New-born Care
HCW Healthcare Worker
287887/2021/DM&A
486 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
6
HIV Human Immunodeficiency Viruses
HRP High Risk Pregnancy
IFA Iron Folic Acid
IMNCI Integrated Management of New-born& Childhood Illnesses
IMR Infant Mortality Rate
I-NIPI Intensified National Iron Plus initiative
IYCF Infant and Young Child Feeding
JSSK Janani Shishu Suraksha Karyakram
JSY Janani Suraksha Yojana
KMC Kangaroo Mother Care
LBW Low Birth Weight
LHMC Lady Hardinge Medical College
LHV Lady Health Visitor
LMP Last Menstrual Period
LSU Lactation Support Unit
MCP Mother and Child Protection
MMR Maternal Mortality Ratio
MO Medical Officers
MOHFW Ministry of Health & Family Welfare
NCD Non-Communicable Disease
NFHS National Family Health Survey
NHM National Health Mission
NIS National Immunization Schedule
NITI National Institute for Transforming India
OGTT Oral Glucose Tolerance Test
OPD Out-Patient Department
ORS Oral Rehydration Solution
PHC Primary Healthcare Centre
PIH Pregnancy Induced Hypertension
POC Proof of concept
PPIUD Postpartum Intrauterine Contraceptive Device
PSMRI Piramal Swasthya Medical Research Institute
287887/2021/DM&A
487 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
7
ReST Representational State Transfer
Rh Rhesus Factor
RO Research Officers
SAANS Social Awareness and Actions to Neutralize Pneumonia Successfully
SAM Severe Acute Malnutrition
SBP Systolic Blood Pressure
SC Sub-Center
SDG Sustainable Development Goals
SIM Subscriber Identification Module
SN Staff Nurse
STG Standard Treatment Guidelines
TT Tetanus Toxoid
U5MR Under Five Mortality Rate
UK-DIT Department for International Trade, Govt of United Kingdom
UN United Nations
UP Uttar Pradesh
VDRL Venereal Disease Research Laboratory test
VHND Village Health, Sanitation, and Nutrition Day
WHO World Health Organization
287887/2021/DM&A
488 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
8
Acknowledgements
The HAIC Study represents a massive collective effort of several teams who persevered through several
obstacles and successfully completed the study despite several unforeseen challenges, COVID-19 being the
foremost. The willingness of each member to go the extra mile was due to a keen inclination to contribute to
improving health outcomes with the use of advanced technology grounded in knowledge-based systems.
Sincere thanks are due to the National Institution for Transforming India (NITI Aayog), particularly Prof V
K Paul who envisioned the study and invited us to lead it. Our special thanks are due to Ms. Anna Roy, Mr.
Sabyasachi Upadhyaya and the Panel of Experts from NITI who steered this study and provided expert
guidance and facilitation, that was cardinal to the scientific conduct, smooth administration, Government-
level approvals, and overall quality of this study.
The Principal Investigators would like to place on record their appreciation for the Government of the
United Kingdom and the UK- Department of International Trade (DIT) for sponsoring this study and
generating evidence for the feasibility of Artificial Intelligence (AI) based solutions in public healthcare
settings in India.
We‘d like to thank the Government of Uttar Pradesh and all the functionaries of the District of Bahraich,
who made it convenient to conduct the study at Nanpara, Kaiserganj, Amwahussainpur, Gayghat, Kundsar
and Bhakhla. The support provided by the local administration, despite straining of resources due to COVID
pandemic, is deeply appreciated.
The expanse of this multi-phase study covering both maternal and pediatric illnesses in the context of
community and first level care required a multi-disciplinary team. We are grateful to the entire Research
Team including the Expert Reference Group, the Advisory Group, the Co- Principal Investigators (PIs) and
Research Officers, who took time out from their busy schedules to guide every step of the process and give
valuable clinical guidance for adaptation of guidelines, implementation, and evaluation. We also acknowledge
the inputs received from the multidisciplinary steering group of NITI Ayaog on their valuable inputs on the
possible improvements and technology use in the way forward studies.
We express our deep adulation for the Elsevier team based out of India and UK, whose tireless efforts in
getting the technology up and running, modulating it based on our feedback, and supporting the
implementation of the study, were critical to the success of this project.
This study would not have been possible without the dedicated effort of the members of Piramal Swasthya
Management Research Institute who were involved in the community part of the study. They helped in initial
training of the participants and then hand-held these primary health care functionaries through the study.
They led from the front at the study site, despite the risks of the pandemic, allowing the researchers to
manage this study virtually, which was forced upon us due to pandemic related travel restrictions.
Finally, and most importantly, we express our sincere appreciation for all the Accredited Social Health
Activists (ASHA), Auxiliary Nurse Midwives (ANM), Staff Nurses and Medical Officers from Bahraich
district who participated in the study. They have put in extraordinary efforts during very difficult times and
shown exemplary levels of commitment to adopt an advanced technology like Clinical Decision Support
System (CDSS) to improve the health of their beneficiaries.
We are also thankful to Dr Rajiv Garg and Dr N N Mathur, incumbent Director(s) of Lady Hardinge Medical
College during the course of this study for their approval and support for conducting this important study.
We sincerely hope that the outcomes of this study shall contribute to the development and use of knowledge-
based technologies like CDSS & AI in India. We believe that this intervention can act as a steppingstone to
the wide-scale adoption of Guidelines and evidence-based practices as the country marches to achieve the
Sustainable Development Goals and an overall improvement in the health of every Indian.
Principal Investigators, HAIC Study
Lady Hardinge Medical College and All India Institute of
Medical Sciences, New Delhi
287887/2021/DM&A
489 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
9
July2021
1. Key Stakeholders and Project Team
Group Role Nodal person
Steering Group
(NITI Aayog
Expert
Committee)
Chairman Dr. V.K. Paul, Member (Health), NITI Aayog
Member Dr. Rakesh Lodha, Professor, Department of Pediatrics, AIIMS
Member Mr. Anurag Aggarwal, Director, IGIB – CSIR
Member Dr. Debdoot Sheet, Asst Professor, IIT Kharagpur
Member Dr. P Santanu Chaudhury, Director, IIT Jodhpur
Member Dr. Balram Ravindran, Professor, IIT Madras
Member Prof. Mausam, IIT Delhi
Member Dr. Sudeep Gupta, Director, ACTREC
Member Ms. Anna Roy, Sr Adviser, Frontier Tech Vertical, NITI Aayog
Investigators
and
Expert Review
Group (ERG)
Principal Investigator Dr. Manju Puri, Director Professor& HOD, OBGYN, LHMC
Co-Principal Investigator
Dr. Virendra Kumar, Director Professor& HOD, Paediatrics,
LHMC
Co-Principal Investigator Dr. Varinder Singh, Director Professor, Paediatrics, LHMC
Co-Principal Investigator Dr. Praveen Kumar, Director Professor, Paediatrics, LHMC
Co-Principal Investigator Dr. RM Pandey, Professor and HOD, Biostatistics, AIIMS
Co-Principal Investigator Dr. K. Aparna Sharma, Addl. Professor, OBGYN, AIIMS
Co-Principal Investigator Dr. Shilpi Nain, Professor, OBGYN, LHMC
ERG member Dr. Shubha Sagar. Trivedi, ex-HOD, OBGYN, LHMC
ERG member
Dr. Monika Rana, Director, Directorate of Family Welfare, Govt
of NCT Delhi
ERG member Dr. Harish Pemde, Director Professor, Paediatrics, LHMC
ERG member Dr. Juhi Bharti, Assistant Professor, OBGYN, AIIMS
ERG member Dr.VidushiKulshreshtha, AssociateProfessor, OBGYN, AIIMS
287887/2021/DM&A
490 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
10
Group Role Nodal person
Project Sponsor
NITI Aayog
Ms. Anna Roy, Sr Adviser, Frontier Tech Vertical, NITI Aayog
Sabyasachi Upadhyay, Associate/Research Officer, NITI Aayog
UK-Department of
International Trade
Ed Rose, Edna D‘Souza, Dr. Eisha Anand
UK-Department of
International Trade –
HealthcareUK
Akoto Agyeman, Rasik Tailor, Natalie Bain, Nicola Parry,
May Noonan, Christopher Jordan, Madhukar Bose, Jane Grady
Technology
partner –
Elsevier
UK representative Dr. Robert Dunlop, Clinical Director
India representative
Dr. Ujjwal Rao, Senior Clinical Specialist
Hema Jagota, Director
Project oversight Meera Saini Gupta, Senior Product Sales Manager
Project
Execution
Partner -
Piramal
Swasthya
Management
Research
Institute
Technical Integration
Mr. Devesh Varma, CTO,
NV Lakshmi V, Sr. Manager IT
Field Implementation
Dr. Bhawna Bakshi, State Transformation Manager
Mr. Balmukund Sharma, District Transformation Manager,
Bahraich, UP
Clinical Oversight Dr. Vibhor Kumar, Chief Manager, Quality and Excellence
Project Implementation Research Officers Dr. Jyoti Mishra, Dr. BindiyaPahuja
Writing Group Dr. Manju Puri, Dr. Varinder Singh, Dr. Ujjwal Rao, Meera Gupta
287887/2021/DM&A
491 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
11
2. Introduction
2.1. Background
Maternal and child health are important indicators for evaluating the country‘s effort to provide equitable
access to health and socioeconomic growth. Global trends indicate that maternal mortality rate dropped by
44% worldwide between 1990 to 2015, however, much remains to be done to achieve SDG target of reducing
the maternal mortality ratio below 70 per 1,00,000 live births by 2030
1. As per WHO, it is estimated that
approximately 830 maternal deaths occur per day due to causes related to pregnancy and child-birth related
complications. Out of the total deaths, 99% occur in developing countries, especially in low resource settings
such as rural areas, poorer communities and hard to reach areas
2. Though India has managed to bring down
the MMR from 556 per 1,00,000 live birth in 1990 to 113 per 1,00,000 live births in 2016-2018, it remains one
of major countries with high MMR burden
3. In 2015, India accounted for 15% of all the maternal deaths
4.
Similarly, despite having made significant progress in reducing under-five mortality rate, it is still unacceptably
high in India. As a signatory to meet Development Goals set out by the UN, India is not only committed to
bring down the under-five mortality rate to 25 deaths per 1,000 live births by 2030; but intends to better it
with a declared aim of reaching U5MR of 23 by 2025. Perinatal deaths, Pneumonia, Diarrhoea are most
common causes of deaths to which undernutrition is an important contributor. While the country is making
some headway but there is a wide variation in performance of the various constituent states. As per an
estimate, only 73% of under-five children suspected to be suffering from pneumonia were taken to health
care providers while only 20% of children suffering from diarrhoea received Oral Rehydration Solutions.
Despite significant improvement in institutional deliveries (79%), early initiation of breastfeeding is seen in
only 41%. Wasting, stunting, anaemia is unacceptably high in Indian children
5.
Antenatal Care (ANC) forms one of four pillars of Safe Motherhood Programme which aims to reduce
maternal mortality. The commonly known causes of maternal mortality such as haemorrhage, sepsis,
hypertension, prolonged labour, unsafe abortion, anaemia, etc. are easily preventable through quality of care
during and after birth. Evidence suggest that Antenatal Care (ANC) reduces pregnancy and childbirth related
morbidity and mortality by almost 80% and should begin from the early stages of pregnancy
6. WHO
recommends a minimum of eight ANC visits, ideally at 12, 20, 26, 30, 34, 36, 38 and 40 weeks,with health
promotion including nutrition counselling as one of its important components
7. Regular ANC enables early
detection of high-risk cases, better management of low weight and micronutrient deficient cases, increased
awareness among women regarding pregnancy care and birth preparedness. Indian guideline recommends at
least four ANC visits with first visit preferably in the first trimester
8.
To increase the coverage and access of public health services, the government has started various
programmes focusing on maternal and child health, family planning, and immunization. Few of the significant
initiatives being, introduction of ASHAs in the health systems to act as a link between community and health
facilities, launch of initiatives such as Janani Suraksha Yojana (JSY), a safe motherhood intervention promotes
institutional deliveries to reduce maternal and infant mortalities by providing financial assistance to pregnant
women (below poverty line) agreeing to deliver in government or accredited private health facilities. Janani
Shishu Suraksha Karyakram (JSSK) is an initiative for providing free and zero expense delivery to all pregnant
women delivering in public health institutions. It entitles free transport from home to health facilities, free
drugs and consumables, free diagnostic, free blood, free diet for the duration of a woman‘s stay in the facility
and covers new-born related illness till 30 days after birth. In case of referral, free transport and treatment is
also provided for two ways
9,10.
However, the mere presence of ANC services and initiatives does not guarantee its utilization nor brings
about a positive outcome. Various reasons exist for the gap in ANC coverage such as availability and
accessibility of health services and care providers, socio economic status, education, community awareness
11.
As per NFHS 2015-16, only 58.6% of the mothers in India had ANC check-up in their first trimester whereas
mothers who had at least four ANC visits accounted for 51.2%. The percentage of women receiving full
ANC check (at least four antenatal visits, at least one tetanus toxoid (TT) injection and iron folic acid tablets
or syrup taken for 100 or more days) in India is very low at 21%. Anaemia during pregnancy is associated
with increased risk of maternal mortality. Fifty three percent of women in the age group of 15- 49 years are
287887/2021/DM&A
492 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
12
anaemic in India
12. It is pertinent to note that the quality of primary healthcare is severely affected by the
strain on Primary Healthcare Centres (PHC) where the PHC, usually led by one Doctor, is expected to
provide comprehensive primary care for up to 30,000 residents
13. A major need for optimal care provision is
to improve the availability of skilled human resources in health systems. There is a shortage of 10,907 ANMs
as against the required number of 1,84,160 and there is a shortage of 3,773 doctors at PHC as against the
required number of 25,743. A similar shortfall is experienced in the count of other specialists and health
personals. Other challenges in providing quality of care is poor infrastructure, According to Rural Health
Statistics 2018, as of 31st March 2018, India has a shortfall of SC, PHC and CHC by 18%, 22% and 30%
respectively
14. 66% of the Indian population lives in rural areas
15 and is largely dependent on the public sector
for their medical needs. Process assessment for the health service components has shown gaps and needs to
improve service delivery process, especially adherence to service guidelines by providers
16,17. Lack of timely,
quality, and convenient access to healthcare in the vicinity, compels them to travel long distances to seek care
thus adding to out-of-pocket expenditure.
To address all the factors affecting poor coverage of maternal and child health, efforts ought to be directed
towards leveraging existing capabilities to provide better quality of care and services to improve key priority
indicators. One promising strategy involves ‗task shifting,‘ where front-line, non-physician health workers are
delegated some of the tasks traditionally performed by physicians
18. In the setting of HIV/AIDS care, task-
shifting has been shown to improve health outcomes and processes of care
19. In India, there is some evidence
that task shifting CVD risk assessment to non-physician health workers via a simple algorithm can increase
the detection of CVD
20.
Empowering front-line health workers, and potentially the mid-level healthcare providers, with knowledge-
based technology solutions, such as Clinical Decision Support System, based on Indian Guidelines can serve
as a job aid and provide manifold benefits. It will allow the care provider administer care based on a standard
treatment protocol while ensuring an appropriate triaging with seamless continuum of care; reduction of
unnecessary intervention and inappropriate referrals; and, provision of pre-referral or definitive treatment at
point of care.
There was an established need for localized clinical decision support, which has also been previously field-
tested in Cardio-vascular diseases
18. However, the need for transforming Standard Treatment Guidelines
(STG) into active clinical decision support, including local dialects, to ensure adherence to STGs in maternal
and child health, was yet to be fulfilled.
2.2. Arezzo®
Arezzo® is an active clinical decision support (CDS) and pathway technology that empowers personalised
care delivery at patient and population level. The technology supports the authoring and execution of Level 3
"active" clinical guidelines that are made computer interpretable through Declarative Artificial Intelligence
(AI). By integrating with clinical care systems, Arezzo® matches evidence-based guidelines with patient and
disease information and dynamically evaluates best-practice treatment options specific to the patient at that
stage in their care. Arezzo® is owned by Elsevier.
Arezzo® CDS proprietary software consists of the following components:
Arezzo® Composer – for configuration of CIGs content
Arezzo® Bridge – for configuration of display-logic, including patient summaries, referral letters,
and to facilitate data-entry
Arezzo® Performer – the high-performance AI engine that processes CIGs content against patient
data to deliver ongoing recommendations, dynamic order sets, and alerts.
Arezzo® Conductor – middleware component that manages external interfaces. In the pilot project,
Arezzo® Conductor will provide the ReST API that is consumed by the mobile app.
MS SQL database for storing the Arezzo® CDS state files and audit data.
In this study, the Arezzo® CIGs were delivered through an Android-based app that was installed on mobile
tablet devices. The mobile app communicated online with the Arezzo® ReST API and dynamically
287887/2021/DM&A
493 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
13
renderedthe outputs to healthcare workers. Outputs included requests for more data, decisions, and tasks.
The app then sent user inputs back to Arezzo® for processing.
Arezzo® was validated through several studies in the UK, Europe, and New Zealand. The details of these
validation studies can be found in Annexure 1.
2.3. Rationale
Arezzo® canenable transformation of primary healthcare by implementing multiple, merged STGs at the
point-of-care, where the FHWs will only see content relevant to the patient. A longitudinal clinical pathway,
based on the State Agent of Arezzo®, will be generated for every unique beneficiary, starting from her
doorstep. Arezzo® will also record the decisions taken for monitoring compliance and variance from STGs,
to automatically generate data about adherence to STGs.
The platform was envisaged to bridge some of the key gaps and challenges in effective healthcare delivery in
the public healthcare delivery system of India. Some of these include:
a. Inadequate and inappropriate referrals from health outreach
b. Knowledge gap in FHWs
c. Non-availability of local language guidance for FHWs
d. Lack of evidence-based primary health screening
e. Inaccurate operational and clinical data for evaluation of compliance measures
f. Lack of health outreach information in the care pathway, which will be addressed by generating a
unique pathway for a beneficiary which could be integrated with the Electronic Health Record
(EHR) in the future
Arezzo® generates high-quality clinical data sets during usage. Whilst such data lends itself to post-hoc
analytics and machine-learning, algorithms can be used to generate potential new clinical insights, but these
must be validated before being incorporated back into CIGs.
It was expected that a secondary gain from a wide-scale Arezzo® implementation in India will be the
significant increase in district-level data about care delivery, referrals, and outcomes, including the potential to
partially automate NFHS-like national surveys.
3. Study Aim and Objectives
The purpose of the study is to test the feasibility of Elsevier‘s ―Arezzo®‖, with adapted Indian Standard
Treatment Guidelines, used in public health care settings through a Pilot intervention.
The study will focus on the following objectives:
a. To test the validity and accuracy of Arezzo® content after its customization based on Indian STGs
demarcated for various levels of care in maternal and child health.
b. To study the feasibility of ―Arezzo®‖ in primary and secondary healthcare settings.
c. To evaluate the following with the addition of Arezzo® in public healthcare settings:
Usability
Usefulness
Adherence to STGs
287887/2021/DM&A
494 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
14
4. Hypotheses
a. Integration of ―Arezzo®‖ - an active Declarative AI clinical decision support system is usable and
useful to health workers in primary and secondary healthcare settings.
b. Provision of Arezzo® customized for Indian guidelines-based recommendations facilitates
adherence to primary health screening guidelines.
5. Methodology
5.1. Study Phases
The study was carried out in two broad phases. Phase 1 was to digitise and transform the STGs to CIGs for
incorporation into the CDS tool and then to assess the validity (Phase 1a) and the reliability (Phase 1b) of the
of the converted guidelines (from STGs to CIGs) in the CDS tool. Phase 2 was to test feasibility of the tool
in terms of the usability, usefulness, and adherence to STGs with its usein the field for Primary Healthcare
and community healthcare workers (CHC, PHC and sub-centre level).
These phases were conducted as per the following timelines:
Preparatory phase: Content conversion and incorporation into CDS (Nov 19-Feb 20)
Phase 1 a: ContentValidity - March 2020
Phase 1b: Content Reliability - April through September 2020
Training for Phase 2 - 28-30 September 2020
Phase 2: Implementation at Study Site (Bahraich, UP) - October 2020 through March 2021
Study data collection - 21-Oct-2020 to 8-April-2021.
Report Writing
Final report submission to sponsors May 2021
Presentation to NITI multidisciplinary steering group and discussions for inputs
HAIC Pilot Project Report (updated post inputs from the steering group)
Figure 1: HAIC Study GanttChart
The phase I was conducted at the department of Obstetrics & Gynaecology and Paediatrics at Lady Hardinge
Medical College while the phase II was conducted in the community settings (Bahraich district in the state of
Uttar Pradesh). The selected district is one of the aspirational districts of NITI and had the presence of a
health support agency, who were taken on board for the study as implementation partner.
Project Milestone31017241815222961320273101724291623306132027411182518152229613202731017243171428512192629233071421284111825181522181522181522181522181522
Arezzo-AMRIT Integration
Ethics Approval
Pediatric Content
Maternal Content
Hindi translation ASHA/ANM
Content Validity (Phase 1a)
Content Reliability (Phase 1b)
Phase 2State ApprovalBaseline SurveysPhase 2Report writing
NITI Aayog update meetings
Final NITI steering group review
MarOctMay Jun
201920202021
AprMarNov Dec Jan FebApr May Jun Jul Aug Sep
1-Jun-21
Nov Dec Jan
26-May-2012-Nov-20
Feb
Arezzo -AMRIT Integration; Installation at NIC
Pediatric Content Conversion
Maternal Content Conversion
Phase 1b
LHMC Ethics ApprovalAIIMS Ethics App
Phase 1a
Translations to Hindi
287887/2021/DM&A
495 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
15
5.2. Target Health Workers
Computer interpretable guidelines (CIGs) using the Arezzo® platform were demarcated, and three levels of
decision-making were programmed for the three rolesviz:Accredited Social Health Activists (ASHA) workers,
Auxiliary Nurse Midwives (ANMs) and Medical Officers (MOs) (Table 1).
Table 1: Target Health Workers
Health Worker Role
Accredited Social Health Activist (ASHA) Front-line health screening
Auxiliary Nurse Midwife (ANM)/Staff
Nurse
Primary care screening and interventions as indicated
Medical Officer (MO) Primary and secondary care
5.3. Constitution of Expert Reference Group
An ExpertReview Group (ERG) of Obstetricians and Pediatricians was constituted for this study (Section 2 –
Key Stakeholders) to provide their expert opinion throughout the duration of the study. The Principal
Investigators (PIs) and co-PIs led the ERG and provided directions on the design and execution of the study
as well as the analysis of the study results. Two Research Officers were also onboarded to help conduct the
study. Elsevier incorporated the STGs into the Arezzo application such that they were demarcated for
various levels of health service providers i.e.MOs, ANMs and ASHAs with the support of the research team.
5.4. Preparatory Field Visit
In December 2019, team members visited Bahraich, UP to understand the workflow at the primary
healthcare facilities and to gain insights for content conversion and development of the solution.The team
met ASHA, ANMs and MOsat 2 pre-selected Primary Health Centers (PHCs), 1 sub-center, 2 Village Health,
Sanitation and Nutrition Day (VHND) clinics and 1 Community Health Center (CHC). Our implementation
partner, Piramal Swasthya Medical Research Institute (PSMRI) helped facilitate the visit including the
meetings. The salient insights were that CHC had high volumes and was the preferred place to go by the
villagers. The infrastructure of the PHCs was impressive but PHC and the sub-center experienced low
volumes of patients. The PIs were impressed by the knowledge and capability of the ASHA workers and the
ANMs. In the maternal population, incidence of anemia was high, but interestingly there were very few cases
of gestational diabetes. This reflected their practice as they did not routinely do screening for GDM.
5.5. Infrastructure and Resources
Android-based tablets with 4G internet connectivity and a pre-installed instance of the AMRIT (Accessible
Medical Records via Integrated Technologies) android-based mobile app, developed by the implementation
partner PSMRI (Piramal Swasthya Management Research Institute), that was integrated with Arezzo®. This
device was used by the Research Officers and the selected health workers enrolled in the study.
5.6. Selection and transformation of Guidelines
As a pilot project to test the hypotheses 10 conditions, 5 each from maternal and child health (Table 2) were
initially finalized through a consensus of the Expert Reference Group (ERG). These guidelines were then
digitized and transformed for use with Arezzo CIGs.
287887/2021/DM&A
496 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
16
Table 2: Guidelines selected for the HAIC Pilot
Condition Guideline
Maternal Health
1.
Antenatal care (ANC) for
uncomplicated pregnancy
Guidelines for ANC Skilled Attendance at Birth by ANMs LHVs
SNs 2010 MOH GOI
ASHA Skills That Save Lives – Module 6 and 7
2.
Gestational diabetes
mellitus (GDM)
Diagnosis and Management of Gestational Diabetes Mellitus -
Technical and Operational Guidelines, NHM, MOH, GOI, 2018
3.
Hypertension in
pregnancy including
eclampsia
Guidelines for ANC Skilled Attendance at Birth by ANMs LHVs
SNs 2010 MOH GOI
4. Anemia in pregnancy
Anemia Mukt Bharat - Intensified National Iron Plus initiative (I-
NIPI) Operational Guidelines, MOH, GOI, 2018
5. Postnatal Care ASHA Skills That Save Lives – Module 6 and 7
Child Health
6. Feeding
National Guidelines on Lactation Management Centres in Public
Health Facilities
Guidelines for enhancing optimal and young child feeding
practices, MOHFW, GOI, 2013
ASHA Skills That Save Lives – Module 6 and 7
IMNCI package for young infant and child
7. Immunization
National Immunization Schedule (NIS) for Infants, Children and
Pregnant Women, MOHFW, GOI, 2017
8. Pneumonia
IMNCI package for young infant and child
Facility Based Care for Sick Children, NIPI, LHMC
9. Diarrhoea
IMNCI package for young infant and child
Facility Based Care for Sick Children, NIPI, LHMC
10. Care of the sick neonate
Facility Based New-born Care Operational Guideline, MOHFW,
GOI, 2011
Home-Based New-born Care Operational Guideline, MOHFW,
GOI, 2014
IMNCI package for young infant
5.7. Detailed Methodology
5.7.1. Phase I
The validity and reliability of the content was tested at the Obstetrics andGynaecology and Paediatric
departments of Srimati Sucheta Kriplani Hospital and Kalawati Saran Childrens‘ Hospital associated with
Lady Hardinge Medical College, andsupported by co-PIs atthe All India Institute of Medical Sciences, New
Delhi.
The purpose of this phase was to assess the validity and the reliability of the of the converted guidelines
(from STGs to CIGs) in the CDS tool. This was done in two parts:
287887/2021/DM&A
497 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
17
Phase 1a: Content Validity
The PIs tested the various steps and components of the individual standard treatment guidelines
(STGs) converted into computer interpretable format (CIGs). both for content (through static
interface) and logical flow (through web-based tester). After each testing session, feedback was sent
for fixing of errors or improvisation to the Elsevier content transformation team and the content was
re-tested again after amendment by Elsevier. Separate pathways were created for ASHA, ANM and
MO keeping in mind their training and expected skill sets. These were further fine-tuned to offer
actions keeping in mind the place of contact (CHC vs. PHC vs. SC vs VHND vs home visits). At all
times, the fidelity to the guidelines was ensured at the concerned levels of care.
The transformation of paper based STGs into computer CIGs went beyond simple digitization. It
included harmonization of overlapping symptoms from the different guidelines and creating AI for
stepwise optimization and unfolding of the action-oriented outcomes from the entered patient data.
Flow of the guidelines had to be adapted according to the flow of real time conversation and
interaction of the HCW with the patients to elicit relevant history related to various
conditions.Dummy patient data was used to test all segments of the CIGs separately for each of the
target health worker roles.
Once all the subcomponents were tested, the whole program was retested using the entire content
together, for example with complex patient journeys that involved multiple symptoms or conditions
across several STGs. Feedback from the retesting was then used to rearrange the CIGs workflow as
required.
Phase 1b: Content Reliability
Ethics committee approval was obtained prior to the commencement of phase 1b.
Phase1b was also conducted in the controlled tertiary care setting at LHMC, using prospective data.
For this phase, the Research Officers weretrained to use Arezzo® tool, now incorporating the CIG
based on National STGs, through a training workshop. They were given mobile tablets equipped
with Arezzo® tester and internet connection for the trial period.
The system was tested prospectively by Research Officers (equivalent to the rank of MOs and
Nurses) under the guidance of the Expert Review Group clinicians using an inter-rater agreement
between the CDSS based decisions (as arrived by the Research Officers) and clinical assessment and
management by ERG clinician.
After each testing session, feedback was sent for fixing of any remaining errors or for improvisation
to the Elsevier content transformation team and the content was re-tested again after amendment by
Elsevier.
Local language conversion in Hindi for standard treatment guideline content relevant to front-line
health workers (ASHA/ANM) was done in parallel to testing in this phase. The testing was done in
English and then in Hindi for the ASHA and ANM/SN content.
Milestones and course adjustments during Phase I:
The salient milestones of this exercise comprised of:
Identification and correction of errors in the CIG content when the test flow did not match the
source STG content.
Adaptation of CIGs to the context of Indian patients; integrating the various vertical guidelines to
cover co-morbidities and to cover different age groups (infant and young child)
Clarifications of STG content: in certain places the flow on the paper guidelines was not clear
andneeded to be articulated/clarified further.
Requirements for new content that were not contained in the source STG but were required to
ensure the appropriate clinical workflow were fulfilled.For example, the obstetric expert team
provided list of questions for:
o taking the obstetric history by a medical officer,
o estimation of duration of pregnancy when the woman cannot recall her LMP
287887/2021/DM&A
498 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
18
Requirements or reorientation for new content that was considered important to cover service areas
and skill set of the peripheral functionaries but was not covered in the selected STGs.
o The guideline cover on Sick New-born was reoriented to cover the Integrated Management
of New-born and Childhood Illness (IMNCI) based care of the sick young infant instead of
facility-based guidelines keeping in mind the job profile, training and skill sets of the target
health functionaries. In addition, home-based New-born care package – a critical
community New-born care intervention- was also included (tested in the healthy New-born
babies in the postnatal ward).
o Emergency Triage and Treatment (ETAT) component of the Care of the Sick child to cover
for cases other than Pneumonia and Diarrhoea for appropriate assessment and emergency
management and/ or referral was created for MOs.
o Developmental assessment for children was included to completely harmonize with the
Mother and Child Protection (MCP) Card.
Harmonization and updating with new guidelines that have been published since the STGs were
created
o For example, the recently published paediatric guideline on assessment and management of
respiratory disease
o Guidance on newer contraceptives: DMPA, Centchroman and PPIUD
Expert opinion on managing the relationships between multiple CIGs in the same patient for
prioritization of treatment
o For example, hypertension in a pregnant woman is more important to manage urgently
compared with anaemia that is not severe.
Information about service provision within the target health district (Bahraich)
o For example, gestational diabetes screening with oral glucose loading test is not performed
routinely
o Also, information about role-based activities enabled the STG content to be separated by
role in the CIGs
In addition, the study plan was impacted severely by the reorganisation of the hospital services due to
COVID. For a significant duration of time, the OPDs had been closed and even when they opened in a
phased manner, the interactions were affected due to the users not willing to stay in the hospital for any
longer than their primary needs for the visit. Travel related restrictions during lockdown, issues related to
worker safety and similar challenges made us make amendments to study plan and the phase 1b was
conducted using the case records of the patients under treatment instead of shadowing of real life OPD
interaction as was originally planned. Many more interactions than the originally planned were done during
this forced extension to extensively validate the digitised care pathways.
Overall, the Principal Investigators of both, the maternal and paediatric teams, met several times face to face
and virtually to review the content. In addition, sub-component testing was also done individually by the PIs
and research officers. The total number of hours spent collectively by the PIs for content conversion and
phase 1 review was over 500 hours. The testing case mix for Phase 1 is detailed in Annexure 4.
Phase 1 study report was submitted and presented to the Expert Committee of NITI Aayog, whose express
approval along with the Ethics Committee approval met the pre-requisites for kicking off Phase 2.
5.7.2. Phase II
This phase involved testing of the feasibility of implementing Arezzo® in primary and secondary healthcare
settingsin the district of Bahraich in Uttar Pradesh, India. The study-site selection was based on discussions
and consensus with NITI Aayog and PSMRI, the project implementation partner.Two CHCs of Kaiserganj
and Nanpara and their associated PHCs were selected,based on feasibility, for the study.
287887/2021/DM&A
499 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
19
Study Participants
50 health functionaries were approached to request voluntary participation in the study. One MO did not
consent to participate in the study. Remaining 49,whoparticipated in the study belonged to the following
cadres: Medical Officers: 8;Staff Nurses:7;ANMs: 14: and ASHAs: 20. Over the course of the study three
ASHAs withdrew from the study for personal reasons and were replaced by 3 other ASHAs from the area.
These changes occurred during the months of November and December‘ 20. From January‘ 21 onwards
there was no further change in the participants. Of the eight MOs enrolled, three did not actively participatein
doing the cases in the digital mode and evaluating the CDSS application. As a result, they also did not
provide their feedback in the final scorecard. A baseline survey was conducted to document the work profile,
language and smart mobile use proficiency and training of the participants.
Table 3: HAIC Study Participants
CHC Nanpara Kaiserganj
Medical Officers 2 3
Staff Nurse 2 2
PHC Amwahussainpur Gayghat Kundasar Bhakhla
Medical Officers 1 1 0 1
Staff Nurse 1 2 0 0
ANMs 4 3 3 4
ASHAs 5 5 5 5
Training
Astravel to the field was severely impacted due to the COVID-19 lockdown, training was conducted virtually
by the PIs, the research officers and Elsevier team. The field staff of PSMRI provided the handholding and
training of the staff on the ground. Training was done over three days from 28-30 September 2020. Initially,
the training was conducted only for ASHA and ANMs. Medical Officers were introduced to the tool later in
November 2020.
Training Modules:
Study overview
Participant information sheet.
Informed consent
Mobile device tablets distribution.
Hands-on training of AMRIT application and CDSS
After the training, the workers were asked to have familiarisation and run-in with the use of application using
test data for 2-3 weeks. The workers were regularlycontacted and encouraged as well as facilitated to use the
app. The study was officially rolled out on 21-October-2020.
Course of the Phase 2 Study
During the initial phase of the phase 2 roll-out the uptake of the application was slow. Internet
connectivity was a major hurdle in many places. Another challenge was related to adoption of the
AMRIT mobile application. The initial plan was to roll out AMRIT one month before Arrezo
clinical decision support system (CDSS) to facilitate familiarization of the HCWs with the app and
the registration of the beneficiaries residing in the area. However, these timelines did not materialize
due to the Covid-related lockdown and AMRIT and Arezzo CDSS were launched at the same time.
As a result, the workers were unfamiliar with usage of the AMRIT, the enumeration application and
were not able to record any interaction with the beneficiary on the digital platform or access CDSS
for the guidance on such an interaction. The internet issues were solved by selecting alternative
287887/2021/DM&A
500 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
20
functional network provider after a one-on-one discussion with the workers. In addition, where the
internet issues persisted, the users were asked to note down the data while in the field and enter it
into the app and complete the CDSS journey later in the day when they reached any area with better
internet connectivity.All these interactions (care seeking journeys) wereanalysed by research officers
and HCWs were regularly contacted by them for easing out any teething troubles and
troubleshootingissues related to technical and/ or content aspect of the programme.
Several issues cropped on integration of Arrezo CDSS with AMRIT, e,g. timing out of the CDSS
server on not receiving timely inputs which were either due to learning curve or slowness of the
connection or due to bugs arising in either of the two programmes. These issues were flagged,
analysed for the cause and resolved in a systematic manner. Both the teams - Elsevier as well as
PSMRI IT team supporting AMRIT - had to work in tandem to sort these.
One month after the roll-out, the research officers and Elsevier team visited the field. This visit
occurred from 22–25 November 2020. During this visit, many training gaps emerged. We found
that the relatively fast-paced virtual trainings had prevented the workers from truly grasping the
concept of the CDSS application and how it was to be used. This visit, therefore, was utilised for
supervised interactions, identification of problem areas and individual feedbacks and reinforcements.
Joint sessions were also held in smaller groups where all the identified problem areas were explained
to the workers.
One of the key outcomes of the app and CDSS tool was to allow identification of high-risk patients,
appropriate referral and step-up care. This required mapping the continuum of care from ASHA to
ANM to MO. However, the functionality to―Get Data‖from one worker to the other inline superior
had problems e.g.ANM was not able to see the beneficiary details of the beneficiary registered by the
ASHA and likewise the MO was not able to view beneficiary details that the ANM or ASHA had
entered. PSMRI technical team worked hard to resolve the integration issues and it required a lot of
fixing, re-fixing and time to get sorted. Each build helped to make the AMRIT application more
stable and the integration more reliable. In all, three successive app builds were released in an
evolutionary Agile approach. Each time a new built was released all the tablets had to be physically
collected from each worker from the field and brought to head office of Piramal at Lucknow, to be
updated and then redistributed. This was a time-consuming exercise.
A final buildincorporating all the desired enhancements was released beginning February2021.
Starting from 7th February 2021, the journeys completed by the workers were analysed almoston
adaily basisand feedback communication with the workers was continued.
Anonymous online surveyswere launched to get feedback from all the users with regards to the
usability and usefulness of the application twice in phase II (one during the 8-18 March ‗21 and
another final one during the first week of April 21. These two surveys had identical questions but
were repeated as the assessment of the usefulness and usability in first survey could have been
affected as the app was not fully stabilized in the initial weeks and many workers who were slow to
adopt the technology had not used the app often enough. We hypothesized that the second survey
which was nearly 8 weeks after the release of the stabilized version of the app wouldallow to get a
better more realistic assessment of its usefulness and usability. The trend of app usage and the access
to STG supported this hypothesis as most of the incomplete journeys (recorded patient interactions)
were mainly seen at the start of the study due to user hesitation, learning curve and technical reasons
like poor or no internet connection, server timing out due to slow inputs/internet, software bugs, etc.
As the questionnaire design of the score card did not allow any detailed understanding of the users‘
view, we invited all of them to participate in smalllocation-based focus group discussions (FGD). It
helped to get detailed insights into their feedback on theusability and usefulness of the solution, its
limitations, and possibilities for future benefits.
46 out of 49 participants responded to the survey. 3 medical officers were not able to respond and
provide their feedback. The following section only details the responses from final survey while the
findings of the midterm survey are detailed in annexure 5. The focus group discussions were
executed from 6-10 April 2021 and the principal investigators were intimately involved in each of the
six focused group discussion sessions that have been detailed in Section 5.1.3.3.
287887/2021/DM&A
501 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
21
The final report hasused parsed data,as compared to the interim report submitted in March 21,
excluding duplicate and/ or unlinked entries forced due to anonymization required for the CDSS
tool, in view of data privacy rules.
The overall process is described in the following flowchart (Fig 2):
Figure 2: HAIC Study Flow-Chart
Sampling
Based on a study and field review by the Principal Biostatistical Investigator, a convenience sampling
methodology with a sample size of 3000 journeys was finalized. Because this was a field study, there was no
requirement for collecting a certain number of case journeys per guideline. Workers were asked to register
and run Clinical Decision Support on a minimum of 5-8 cases a week.
Inclusion criteria
Pregnant women seeking antenatal care or postnatal care and Children under 5 years, with any of the
conditions included in the pilot (section 4.6) in contact with a health worker in community setting or
those attending a PHC/CHC
5.8. Outcome Measures
The following outcome measures, based on operational criteria specific to the project, as well as clinical
evaluation criteria for maternal and childhealth were evaluated.
287887/2021/DM&A
502 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
22
Table 4: HAIC Study Outcome Measures
Phase
Outcome
Measure
Type of
Data
Evaluation Criteria Data security
Phase 1a Content validity Qualitative Concordance by PIs Non-identifiable data
Phase 1b
Content
reliability
Qualitative Concordance by PIs
Anonymized patient data
Secure transmission of data
through encryption
Phase 2
Usability
Quantitative
Qualitative
Anonymized
Quantitative score
card surveys
@
FGD
Written informed consent
Usefulness
Quantitative
Qualitative
Anonymized
Quantitative score
card surveys
@
FGD
Written informed consent
STG Adherence Quantitative
Cumulative STG
Adherence Trend
STG Adherence
indicators
Pseudonymized patient data
Secure transmission of data
through encryption
SSL certification
DNS over HTTPS
@Score card-Annexure2
5.8.1. STG Adherence Indicators
The following STG adherence indicators were measured through automated data collection via Arezzo:
Table 5: STG Adherence Indicators
Condition STG Adherence Indicators
Maternal Health
Antenatal care (ANC) for
uncomplicated
pregnancy
1. Documented LMP
2. Documented EDD
3. History taken (specific symptoms, systemic illness, drug intake,
family history of systemic illness etc.)
4. Weight measured
5. Blood Pressure taken
6. Blood sample taken (for Hb)
7. Blood sample taken (for blood group, Rh)
8. Blood sample taken (for VDRL)
9. Urine sample taken (for protein)
10. Urine sample taken (for glucose)
11. Abdomen examined
12. IFA supplements prescribed
13. Tetanus Toxoid prescribed
14. Calcium supplements prescribed
15. Anthelmintics prescribed
16. Counselling done (dietary advice)
17. Counselling done (contraception)
18. Counselling done (violence)
19. Counselling done (rest)
20. Documented referral when indicated
Gestational diabetes
mellitus (GDM)
21. Screening for GDM done
22. History taken for signs & symptoms of hyperglycemia.
23. OGTT done for screen-positive women
287887/2021/DM&A
503 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
23
Condition STG Adherence Indicators
Hypertension in
pregnancy including
eclampsia
24. Documented history (hypertension or pre-eclampsia/eclampsia in
the previous pregnancy or current pregnancy or family history)
25. Two consecutive BP readings taken four hours or more apart if
SBP ≥ 140 mmHg or DBP ≥90 mmHg or more
26. Test for urine albumin in case of high BP
Anemia in pregnancy
27. Documented history suggestive of anemia (pallor, palpitation,
fatigue)
28. Hemoglobin test at each visit
29. IFA supplements – one tablet twice a day
Postnatal care
30. Iron prescribed
31. Contraceptive advice given
32. Nutrition advice given
33. Breast feeding advice given
Child Health
Feeding
34. Breast feeding within one hour of birth
35. Counselling done on lactation support, breast feeding, IYCF and
KMC (as part of LSU requirements), on complementary feeding
and done on feeding and recommendations during sickness and
health
Immunization
36. Documented immunization history as per National Immunization
Schedule (NIS)
37. Recommendations for vaccination as per NIS
Pneumonia
38. Documented assessment (signs of pneumonia and pneumonia
severity)
39. Documented referral when indicated
Diarrhoea
40. Documented history (feeding, SAM, diarrhoea, blood in stool)
41. Documented assessment (persistent diarrhoea, dysentery, signs of
dehydration)
42. ORS packet prescribed when indicated
43. Zinc prescribed when indicated
44. Documented referral when indicated
45. Antibiotics prescribed
Care of the sick neonate
46. Documented history (preterm, LBW, intrapartum/post-partum
complications)
47. Documented assessment (ETAT)
48. Documented assessment (feeding, crying, breathing, icterus,
pallor, cyanosis, abdominal distension, neck rigidity)
49. Documented referral when indicated
General
50. Documented birth weight
51. Weight measurement on every visit
52. Temperature measurement
287887/2021/DM&A
504 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
24
6. Results
6.1.1. Base-line Survey
A baseline survey was conducted to understand the current practices of these field workers.
While Hindi language proficiency was 100% across all three groups, it was evident that English
proficiency was much higher in ANMs as compared to ASHAs, with 100% proficiency in MOs.
All three cadres of health workers were familiar with STGs and were trained on newly published
guidelines.
The comfort level with mobile devices was highest in ANMs followed by MOs and then ASHAs.
However, none of them used any mobile devices for any routine work.
The perceived benefit of CDSS was highest for ASHAs, followed byMOs, and thenANMs.
Figure 3: Baseline Survey Results
Further details of the baseline survey are contained in Annexure 3.
6.1.2. Phase 1 Results
Phase 1a: Content Validity
All the content related to common illnesses and conditions selected for the pilot study could be
accurately digitized with separate pathways for ASHA, ANM and MO keeping in mind their training,
skills and job profile.
The AI could allow accurate stepwise optimization and unfolding of valid and accurate action-
oriented outcomes from the entered patient details, keeping in context the place of contact (PHC vs.
CHC vs.VHND vs. home visits).
The fidelity to the guidelines was maintained at all times and at all levels of care. It worked equally
well with complex patient journeys that involved multiple symptoms or conditions across several
STGs.
Phase 1b: Content Reliability
Reliability Testing was done on 439 cases (208 maternal and 231 paediatric) as detailed in the Table
below. It coveredmore than the proposed cases that were to be tested for both Maternal and
H I N D I
P R O F I C I E N C Y %
E N G L I S H
P R O F I C I E N C Y %
S T G A W A R E N E S S
%
C O M F O R T L E V E L
W I T H M O B I L E
D E V I C E S %
P E R C E I V E D
B E N E F I T O F
C D S S %
100
15
100
70
30
100
81
100
81
5
100100100
78
11
Baseline Survey
ASHA ANM MO
287887/2021/DM&A
505 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
25
Paediatric guidelines covering the different health functionary roles. Multiple, relevant journeys were
run on these cases, separately for each of the roles (ASHA, ANM and MO). In total, we evaluated
963 maternal and 2172 paediatric journeys. The details of the cases are given in Table 6 below and
the breakdown of journeys is provided in Annexure 4.
Table 6: Phase 1 Reliability Testing Journeys
Each Case completed for all the three roles ASHA, ANM and Medical Officer.
Maternal Cases (N=208)
ANC-without any complications 63
ANC with 1 or more complications
Anemia in Pregnancy
Preeclampsia/Hypertension/Eclampsia
Gestational diabetes mellitus
105
42 *
38 **
33 ***
Post-natal 40
Pediatric Cases (N=231)
Feeding and Immunization alone 33 #
Pneumonia
Pneumonia alone
With feeding and immunization
24 ##
11
13
Diarrhoea
Diarrhea alone
Diarrohea w SYI
With sick child
With pneumonia
With feeding and immunization
76 ###
31
10
6
5
24
Care of the sick child
Care of the sick child alone
With feeding and immunization
44
22
22
Sick Young Infant 32
New-born Care 22
* Includes women with anaemia alone and also women with anaemia and or
preeclampsia/ GDM
** Includes women with preeclampsia alone and also women with preeclampsia and or
anaemia/ GDM
*** Includes women with GDM alone and also women with GDM and /or anaemia/
preeclampsia.
# In total, 92 cases each were done for Feeding and Immunization (rest depicted
elsewhere in the table as and when they were overlapping with other guidelines; the
feeding and immunisation does not get assessed in severe cases as referral takes
precedence over these assessment)
## Includes children with Pneumonia alone and also pneumonia with feeding and
immunization.
### Includes children with diarrhoea alone and diarrhoea for sick young infant, sick
child, pneumonia and diarrhoea with feeding and immunization
287887/2021/DM&A
506 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
26
At the end of phase 1b, all the maternal and paediatric conditions for whom content conversion was
done and AI was created have been tested. The CDSS content on the integrated software (Arezzo-
Amrit app) was tested and retested on a sufficiently large dataset (Table 6) and all the bugs identified
were removed. The software has achieved the capacity to effectively and seamlessly providing
individualized intervention or management solutions with complete reliability, based on the
beneficiaries‘ case details provided.
Furthermore, this fine-tuning of content was coupled with further need based enhancements.
Notably the application is also able to do the following:
o create necessary clinical summaries when beneficiaries are referred from the ASHAs to
ANMs and from front-line health workers to the medical officers.
o covers the early development screening for children up to 3 years.
o supplemented with the addition of images and harmonized with the content of the mother
and child protection card introduced recently in the functioning of front line workers.
The ERG led by PIs reviewed the journeys and agreed that the converted guidelines were sufficiently
and satisfactorily tested for validity and reliability according to the pathways outlined. STG content
was found to be fully transformed and supplemented with more recently published
recommendations, where deemed essential by the experts. The content was further harmonized and
fine-tuned across guidelines using advisory group opinion.
The ERG led by PIs had concordance on the sufficiency of testing to cover majority of the likely
case scenarios for the selected conditions having significant impact on maternal and child health in
the primary setting.
Further, there was concordance on the ability of the configured and optimized version of the
Arezzo® Clinical Decision Support Systems integrated with AMRIT to reliably and accurately
provide individualized treatment / care decision, appropriateto the role of the health functionary and
in complete fidelity with the adopted National guidelines.
6.1.3. Phase 2 Results
In Phase 2 the computer interpretable guidelines and the CDSS were available to use by the health workers
through the app, we used a mixed method approach and collected both quantitative data as well as qualitative
data. The user‘s experience was assessed quantitatively using anonymised survey using scalar scale score cards.
Adherence to the standard guidelines was assessed quantitatively through backend usage data.
Qualitative data was collected through focus group discussions done in smaller batches after the score card
surveys were submitted to delve deep into the reasons for the scoring and get insights into challenges and
desired modifications, if any.
6.1.3.1. Quantitative Scorecard Surveys Analysis
Statistical analysis of the end of study scorecard results was performed using KNIME Analytics Platform
(v4.3.2) and Microsoft Excel for Microsoft 365 MSO and detailed below (Fig 4).
The cumulative positivity rate (rated as agree and strongly agree) for questions related to
Usabilitywas76.09%(Mean Likert Score 3.99; 95% CI: 3.41 – 4.57).
The cumulative positivity rate (rated as agree and strongly agree) for questions related to
Usefulnesswas 80.43%(Mean Likert Score 4.03; 95% CI:2.38– 4.77).
The cumulative positivity rate (rated as agree and strongly agree) for questions related to Overall
Satisfaction was 76.09%(Mean Likert Score 4.10; 95% CI: 3.80 – 4.39).
287887/2021/DM&A
507 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
27
Figure 4: Quantitative Scorecard Survey Results Summary(Phase 2)
The slightly higher cumulative positivity rate for Usefulness indicates the participants inclination towards the
features that they found valuable in their routine tasks, but this wasn‘t necessarily supported by an equal
inclination towards the ease of use of the application, reflecting on its Usability, that wasinitially impacted by
factors relating to technology, network availability and integration with the PSRMI AMRIT app. FGDs
provided further insight into the reasons for these differences and are discussed later.
Usability
The cumulative positivity rate where respondents agreed or strongly agreed on the Usability of CDSS-
integrated mobile was 76.09%(Mean Likert Score 3.99; 95% CI: 3.41– 4.57) with statistically significant
responses to all questions on usability (p <0.05;one-sample t-Test).
The cumulative positivity rate and Mean Likert Scores for each question on Usability were as follows:
The mobile application iseasy to use: 89.13%and 4.33(95% CI: 4.08 - 4.57)
The necessary information and summariesare organized and displayed in a logical manner: 86.96%
and 4.13 (95% CI: 3.91 - 4.35)
I can accomplish my tasks more easily: 73.91% and 3.98 (95% CI: 3.66 – 4.30)
Recording patient data and navigating through the workflow is seamless: 73.91% and 3.91 (95% CI:
3.61– 4.21)
Feel comfortable using it in the social or community setting: 69.57% and 3.85(95% CI: 3.55– 4.15)
It integrates and fits easily into my daily routine: 63.04%and 3.72(95% CI: 3.41 – 4.03)
There was also a statistically significant variance between different groups for the question ―It integrates and
fits easily into my daily routine‖ (p <0.05; One-way ANOVA) with mean Likert Scores for four groups,
showing a higher positivity rate amongst ASHAs compared to the other groups, as follows:
ASHA: 4.2 (95% CI: 3.75 – 4.65)
ANM: 3.21 (95% CI: 2.58 – 3.86)
Medical Officer: 3.20 (95% CI: 2.64 – 3.76)
Staff Nurse: 3.71 (95% CI: 2.83 – 4.59)
There was no statistically significant variance between different groups of ASHAs, ANMs, Staff Nurses and
Medical officers for the other questions on usability.
0%
20%
40%
60%
80%
100%
Usability
Usefulness
Overall Satisfaction
Usability, 76.09%
Usefulness, 80.43%
[CATEGORY
NAME], [VALUE]
USERS WITH
CUMULATIVE
POSITIVE SCORE
RATING
Cumulative Positivity Rate
287887/2021/DM&A
508 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
28
Figure 5: Usability Survey Results
Figure 6: Usability Survey Stacked % Results
5
8
4
4
2
6
9
9
7
4
4
16
17
19
20
26
19
18
12
13
14
14
22
051015202530354045
Icanaccomplishmytasksmoreeasily
Itintegrates/fitseasilyintomydailyroutine
Ifeelcomfortableusingitinsocial/communitysetting
Recordingpatientdataandnavigatingthroughthe
workflowisseamless
Thenecessaryinformationandsummaryisorganizedand
displayedinalogicalmanner
Itiseasytousethismobileapplication
Usability (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
0%
20%
40%
60%
80%
100%
Usability % (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
287887/2021/DM&A
509 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
29
Usefulness
The cumulative positivity rate where respondents agreed or strongly agreed on the Usefulness of CDSS-
integrated mobile was 80.43% (Mean Likert Score 4.03; 95% CI: 2.38– 4.77) with statistically significant
responses to the following questions on usefulness (p <0.05;one-sample t-Test) with the positivity rate
and mean Likert Scores as recorded below:
It gives me more information on what to advise patients: 97.83 % and 4.59(95% CI:4.40– 4.77)
It helps me remember all diagnostic procedures to be advised: 95.65%and 4.52(95% CI:4.33–4.72)
This tool helps me administer the right treatment and the right drug at the right dose: 93.48% and
4.48(95% CI:4.25– 4.70)
It enables me to determine which patients should be referred: 91.30% and 4.30(95% CI:4.12– 4.49)
I am more confident when I talk to patients about their conditions and recommendations: 86.96%
and 4.33(95% CI:4.1–4.55)
It is useful in doing my daily tasks: 82.61%and4.20(95% CI:3.98– 4.41)
By using this application, my work skills have increased, and I am able to accomplish my tasks
quickly: 73.91% and 4.09(95% CI: 3.81 – 4.37)
Two questions on Usefulness had statistically insignificant responses (p >0.05;one-sample t-Test) with the
positivity rate and mean Likert scores as follows:
The quality of my interactions has reduced due to the time spent in entering data and reading
instructions from the tool: 58.70% and 2.80 (95% CI: 2.38 – 3.23)
I often need to override the suggestions made by the tool: 43.48% 2.96 (95% CI: 2.53 – 3.38)
In addition, there was a statistically significant variance between different groups of health functionaries for
the question ―I often need to override the suggestions made by the tool‖ (p <0.05; One-way ANOVA)
with mean Likert Scores for four groups, showing a higher negative rate amongst ASHAs compared to the
other groups:
ASHA: 2.25 (95% CI: 1.63 – 2.87)
ANM: 3.57 (95% CI: 2.8 – 4.35)
Medical Officer: 3.20 (95% CI: 2.16 – 4.24)
Staff Nurse: 3.57 (95% CI: 2.17 – 4.97)
These issues were further delved into during the FGD sessions to gain insights into when and why the users
felt the need to override the suggestions made by the tool. During the facilitation of app use and shadowing
of case interactions, we had noticed that many ASHAs and ANMs had some difficulty in correctly
interpreting an indirect or negative question. The response to these questions may have similarly got affected.
There was no statistically significant variance between different groups of ASHAs, ANMs, Staff Nurses and
Medical officers for the other questions on usability.
287887/2021/DM&A
510 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
30
Figure 7: Usefulness Survey Results
Figure 8: Usefulness Survey Stacked % Results
9
7
3
8
1
1
1
2
14
8
9
2
5
1
1
4
5
21
15
19
16
18
17
16
24
10
17
19
8
29
22
27
27
18
10
0 51015202530354045
It is useful in doing my daily tasks
By using this application, my work skills have
increased, and I am able to accomplish my tasks quickly
The quality of my interactions has reduced due to the time
spent in entering data and reading instructions from the
tool
It gives me more information on what to advise patients
I am more confident when I talk to patients about their
conditions and recommendations
It helps me remember all diagnostic procedures to be
advised
This tool helps me administer the right treatment and the
right drug at the right dose
It enables me to determine which patients should be
referred
I often need to override the suggestions made by the tool
Usefulness (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
0%
20%
40%
60%
80%
100%
Usefulness % (N=46)
Strongly Disagree ?????? Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
287887/2021/DM&A
511 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
31
Overall Satisfaction
The cumulative positivity rate where respondents agreed or strongly agreed on the Overall Satisfaction with
of CDSS-integrated mobile was 76.09%(Mean Likert Score 4.10; 95% CI: 3.80 – 4.39) with statistically
significant responses to the two constituent questions(I would like to continue using this tool for my daily routine
and work AND Overall, I am satisfied with the CDS tool)(p <0.05;one-sample t-Test).
There was a statistically significant variance between different groups on the above two questions relating to
the overall satisfaction (p <0.05; One-way ANOVA) with mean Likert Scores for four groups, showing a
higher positive rate amongst ASHAs and Staff Nurses compared to the other groups:
I would like to continue using this tool for my daily routine and work: Overall Positivity rate
and mean Likert Score:73.91% 4.04 (95% CI 3.80 – 4.29)
o ASHA: Positivity rate: 100%4.50(95% CI:4.26 – 4.74)
o Staff Nurse: Positivity rate: 57% 4.29 (95% CI:3.41 – 5.17)
o ANM: Positivity rate: 50% 3.50(95% CI:3.06 – 3.94)
o Medical Officer: Positivity Rate: 40% 3.40(95% CI: 2.72 – 4.08)
Overall, I am satisfied with the CDS tool: Overall positivity rate: 78.26% 4.15 (95% CI 3.91 –
4.39)
o ASHA: Positivity Rate: 100% 4.55(95% CI:4.31 – 4.79)
o Staff Nurse: Positivity Rate: 85.71% 4.43 (95% CI: 3.38 – 5.48)
o Medical Officer: Positivity rate: 60% 3.60 (95% CI: 2.92 – 4.28)
o ANM: Positivity rate: 50% 3.64(95% CI:3.21 – 4.07)
Figure 9: Overall Satisfaction Survey Results
It was intriguing that the proportion of users who were over all satisfied was higher than those who would
like to continue using the tool. We delved into the reasons thereof in the FGDs.
Correlation
We were aware of the possibility of possibly receiving some random responses instead of a thought out
answer from the users. We therefore tried to find correlation between the responses to various questions and
see whether the correlations were logical.Certain responses to questions on Usability, Usefulness and Overall
Satisfaction had statistically significant correlations (p <0.05; Spearman‘s Rank Correlation test).
“It integrates/fits easily into my daily routine” had a strong positive correlation with “I would like to
continue using this tool for my daily routine and work”(ρ=0.70)
“It integrates/fits easily into my daily routine” had a strong positive correlation with “Overall, I am satisfied
with the CDS tool” (ρ=0.65)
0
5
10
15
20
Iwouldliketocontinueusingthistoolformy
dailyroutineandwork
Overall,IamsatisfiedwiththeCDStool
11
11
9
19
18
15
18
Overall Satisfaction (N=46)
Disagree ?????? Neutral ?????? Agree ?????? Strongly Agree ??????
287887/2021/DM&A
512 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
32
“By using this application, my work skills have increased, and I am able to accomplish my tasks quickly”had a
strong positive correlation with“I would like to continue using this tool for my daily routine and work”(ρ =
0.62)
“I can accomplish my tasks more easily quickly”had a strong positive correlation with ―Overall, I am
satisfied with the CDS tool‖(ρ= 0.61)
“I often need to override the suggestions made by the tool”had a moderate negative correlation with“By using
this application, my work skills have increased, and I am able to accomplish my tasks quickly”(ρ =-0.50)
“I often need to override the suggestions made by the tool” had a moderate negative correlation with “It
integrates/fits easily into my daily routine(ρ =-0.42)
6.1.3.2. STG Adherence
Two quantitative methods of STG adherence that were adopted included a cumulative STG adherence trend
based on an analysis of completed journeys and specific STG adherence indicators for each of the 10 clinical
conditions included in the study.
Cumulative STG adherence trend (based on proportion of completed CDSS journeys)
Cumulative STG adherencerefers to a measure of the overall use of CDSS, regardless of the specific
STGs content that was processed by the CDSS engine. When a user completes a CDSS journey then
all necessary enquiries, decisions and recommendations that are part of the transformed content
associated with the merged STGs is completed. Abandoning a journey midway may be considered as
non-adherence to STGs.
From 21 October 2020 through 8 April 2021, 4460 beneficiary interactions had a clinical decision
support session started via the mobile app. Of these, 3995(90%) of the CDSS journeys started were
completed by either an ASHA, ANM, nurse or MO. CDSS journeys were not completed in 10% of
cases. At the start of the study, many journeys were incomplete possibly due to user hesitation, the
learning curve and technical reasons like poor or no internet connection, or software bugs, etc but as
the study progressed the numbers of completed journeys increased. The issues were monitored in
real-time by the Project Team to facilitate prompt intervention, which resulted in more journeys
being completed once the initial technical hitches (prior to December 2020) were sorted (Fig 10).
Figure 10: STG Adherence Trend for completed journeys(Oct-2020 to April 2021)
19-Oct26-Oct02-Nov09-Nov16-Nov23-Nov30 N0v07-Dec14-Dec21-Dec28-Dec04-Jan11-Jan18-Jan25-Jan01-Feb08-Feb15-Feb22-Feb01-Mar08-Mar15-Mar22-Mar29-Mar05-Apr
Journeys Completed117 8 1 0 7623995214825748534527214543257550834614445267118
Journeys Not completed113 8 3 0 84870304013132629121523403017115 2 6 2
Total Journeys 2210164 015110109125254957051137428416045561553836315550287720
% Completion of Journeys50%70%50%25%0%47%56%36%76%84%86%81%95%92%96%91%95%93%94%95%93%90%93%92%90%
50%
70%
25%
47%
56%
36%
76%
86%
81%
92%
96%
91%
95%
90%
92%
0%
25%
50%
75%
100%
0
200
400
600
Primary
Axis
-
Number
of
Journeys
Secondary
Axis
-
%
Completion
of
Journeys
Stable App release
Stable App releaseCOVID-19
vaccination drive
Early App release
One to one
retraining
287887/2021/DM&A
513 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
33
Adherence to Specific STG Adherence Indicators
A more detailed analysis of adherence to specific STG indicators was also carried out from analyzable
journeys in the data-set where all the information could be accessed.
Tables 7and 8providesthe STG adherence indicator clinical measures that werecomputed through SQL
queries on the Arezzo data base.
Summary of analysis:
STG adherence for maternal indicators was 77.28% (95% CI: 65.0 – 89.6) with statistically
significant mean adherence (p<0.05; one sample t-Test).
STG adherence for paediatric indicators was 74.96% (95% CI: 58.1 – 91.8) with statistically
significant mean adherence (p<0.05; one sample t-Test).
Interpretation of STG Adherence measures:
Majority of the indicators had 100% adherence. However, there were indicators that had less than
5% adherence. Majority of these were a consequence of either practice gaps at the ground level or
due to a lack of resources. E.g. Blood sample collection for blood grouping was 3%, but Urine
Samples were regularly collected for urine proteins (87%) and sugar (83%), due to entrenched
practices followed by the ANMs. Two consecutive BP readings are not taken four hours or more
apart if SBP > 140 mmHg or DBP >90 mmHg, again indicating a practice gap.
Another instance of practice gap in paediatrics is weight and temperature measurement on every visit
which is as low as 16%.
Screening for gestational diabetes mellitus was as low as 1% with zero OGTTs done for screen-
positive women. This has a two-fold causation – firstly, as per the medical officers and nurses, there
are virtually no cases of GDM in Bahraich, but interestingly there were not dipstick tests or OGTT
packs available for screening, which could also signal some level of undetected GDM at the study
site.
Another prominent indicator with low adherence was ―Documented referral when indicated‖, which
corroborates with the on-ground findings that indicated low referrals. The research officers did try to
do a root-cause analysis of this finding but the analysis was not able to be performed in real time, and
thus some of these cases may have been missed. This is an issue that will need to be reviewed in
more detail in future studies.
In Pediatrics, earlier a single developmental delay was triggering a referral with a very low threshold,
and it should have been set to only a high-risk level. This was subsequently fixed in the content and
unindicated referrals reduced significantly. However, this did impact the overall STG adherence
indicator related to referrals.
Details of secondary data analysis over the month of February of the STG adherence is contained in
Annexure 6.
287887/2021/DM&A
514 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
34
Table 7: STG Adherence Indicators for Maternal Health
* Eligible visits refer to the total number of visits that would have required STG adherence for each indicator. Not
all visits required STG adherence for every indicator, depending on the STG recommendation and the role. For
example, ASHAs were not expected to take blood pressure recordings on any visit. This is why the number of
eligible visits varies throughout the Table.
** Marked as green because not all women were able to provide an LMP but the question was always asked.
*** 14 referral recommendations were not followed because: user overrode the recommendation (4); family
reasons (4); transport issues (1); other reasons – not specified (9). Antenatal care (ANC) for uncomplicated pregnancy
1Documented LMP **1042926 89%
2Documented EDD10421042 100%
3History taken (specific symptoms, systemic illness, etc) 10421042 100%
4Weight measured930930 100%
5Blood pressure taken930930 100%
6Blood sample taken (for Hb)889829 93%
7Blood sample taken (for blood group, Rh)68318 3%
8Blood sample taken (for VDRL)889710 80%
9Urine sample taken (for protein)951827 87%
10Urine sample taken (for glucose)889741 83%
11Abdomen examined930930 100%
12IFA supplements prescribed1012967 96%
13Tetanus Toxoid prescribed766748 98%
14Calcium supplements prescribed889872 98%
15Anthelmintics prescribed897433 48%
16Counselling done (dietary advice)11591154 100%
17Counselling done (contraception)774702 91%
18Counselling done (violence)775773 100%
19Counselling done (rest)11591152 99%
20Referral recommended ***3319*** 58%
Gestational diabetes mellitus (GDM)
21Screening for GDM (blood glucose)88915 2%
22History taken for signs & symptoms of hyperglycaemia 13431343 100%
23OGTT done for screen positive women00 0%
Hypertension in pregnancy including eclampsia
24
History asked (hypertension or pre-eclampsia/eclampsia in any
pregnancy or family history)
719719 100%
25
Two consecutive BP readings taken four hours or more apart if SBP ≥
140mmHg or DBP ≥ 90mmHg
200 0%
26Test for urine albumin in case of high BP2014 70%
Anaemia in pregnancy
27Documented history suggestive of anaemia (pallor, palpitations, fatigue)13431343 100%
28Haemoglobin test at each visit889829 93%
29IFA supplements – one tablet twice a day37176 20%
Postnatal care
30Iron prescribed138136 99%
31Contraceptive advice given124121 98%
32Nutrition advice given124124 100%
33Breast feeding advice given124123 99%
No.STG Adherence IndicatorEligible visits*
No. of visits with
evidence of
compliance
Adherence (%)
287887/2021/DM&A
515 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
35
Table 8: STG Adherence Indicators for Child Health
* Eligible visits refers to the total number of visits that would have required STG adherence for each
indicator. Not all visits required STG adherence for every indicator, depending on the STG recommendation
and the role. This is why the number of eligible visits varies throughout the Table above.
6.1.3.3. Qualitative data: Focused Group Discussions
Following the quantitative surveys, it was deemed critical to uncover latent insights on the usability,
usefulness, and propensity to facilitate adherence to STGs with CDSS through FGDs, that could further
enhance the learnings from the results obtained thus far. All study participants were divided into five groups
based on their role and district for five virtual FGD sessions. Each FGD usually had the members from a
particular cadre so that there was an ease among the participants due to familiarity and no fear of a superior
officer‘s presence. The health functionaries met at a convenient time and place of their choice near their work
area. All sessions were attended virtually by the researchers while the project implementation partner
representatives present on-site facilitated the meeting through virtual platforms. Although these FGDs
externally transpired as informal discussions, the PIs leading the sessions followed a tacit structure that
involved broad questions that further led into more specific areas of enquiry, with equal representation from
all participants. There was a pre-decided checklist used by the investigators, which largely focused on the
same questions as the quantitative survey but intended to unravel the reasoning behind the responses. The
sessions were recorded with informed consent on the assurance of anonymity of individual responses.
The responses received in the meetings were broadly classified around the themes of usefulness and usability
of the tool and have been organised thematically, on the same lines, below. We have recorded the verbatim
comments in Hindi and also translated them into English for easy reckoning. Feeding
34 Breast feeding within one hour of birth34 34 100%
35
Counselling done on lactation support, breast feeding, IYCF and KMC (as part of LSU requirements), on
complementary feeding and on feeding and recommendations during sickness and health
2569 2569 100%
Immunization
36 Documented immunization history as per National Immunisation Schedule (NIS) 2893 2893 100%
37 Recommendations for vaccination as per NIS1799 1799 100%
Pneumonia
38 Documented assessment (signs of pneumonia and pneumonia severity)2798 2798 100%
39 Documented referral when indicated16 5 31%
Diarrhoea
40 Documented history (feeding, SAM, diarrhoea, blood in stool)3080 3080 100%
41 Documented assessment (persistent diarrhoea, dysentery, signs of dehydration) 3094 3094 100%
42 ORS packet prescribed when indicated26 26 100%
43 Zinc prescribed when indicated26 25 96%
44 Documented referral when indicated7 2 29%
45 Antibiotics prescribed 47 12 26%
Care of the sick neonate
46 Documented history (preterm, LBW, intrapartum/post-partum complications)356 356 100%
47 Documented assessment (ETAT)62 62 100%
48
Documented assessment (feeding, crying, breathing, icterus, pallor, cyanosis, abdominal distension, neck
rigidity)
356 356 100%
49 Documented referral when indicated80 31 39%
General
50 Documented birth weight179 128 72%
51 Weight measurement on every visit3041 492 16%
52 Temperature measurement3161 521 16%
No.STG Adherence Indicator
Eligible
visits*
No. of visits with
evidence of
compliance
Adherence
(%)
287887/2021/DM&A
516 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
36
Usefulness of the tool
We asked the participants as to what was the best thing they liked about the tool to document what were the
most useful features from their viewpoint. The responses are documented below clubbed under common
thematic domains.
It was evident in all the FDGs across all sections of the users that the foremost perceived
benefit was a sense of empowerment due to improved knowledge.
ASHAs and ANMs commonly supported a general perception of augmentation of knowledge as
stated by two of the participants:
o ―ज़्यादाजानकारीममलीींहै‖(Got more information) and ―App सेकाफी knowledge ममलतीहै”(The app provides
a lot of knowledge),
o more specific elements of knowledge, as mentioned by another participant
―बहुतसारीजानकारीजोहमेंनहीींथीवहइस app के ज़ररएमालूमहोतीहै, जैसेकीब�ेमजसतरहबढ़तेहै,
मकसउम्रमेंउन्हेंबोलनाहै, चलनाहै, जोइसके Questions मेंआताहै, वहहमेंपहलेनहीींपताथा(A lot of information that
we were earlier not aware of, we got to know through this app, such as the way children grow, at what age
they should talk, walk, which comes in the form of questions, that we did not know before).
o There was also a sense of incremental knowledge compared to theearlier state, as suggested
by one participant ―जोपहलेपूछतेथे, उससेextra हैइस app में (This app has extra information than what
we used to ask earlier).
There was a manifest inclination towards the capabilities of the app to serve as an effective
job aide, particularly by ASHAs and ANMs.
o Automatic calculations of dates: One participant mentioned ―गभभवतीममहलाकी LMP से EDD
कापतालगजाताहै (We get to know a pregnant woman's EDD based on her LMP). ―काफीसारीcalculations
अपनेआपहोींजातीहैं” (Quite a few calculations get done automatically).
―इसमेंहमटीकालगानेजातेहैतोअगलीताररकआजातीहै” (If we go for vaccination this gives you the next date) said
another.
o Tips for situation specific and appropriate counselling: The app‘s capabilities to generate
sound counseling and referral advice was also appreciated.
―App जोसलाहबताताहैवोबहुतअ�ाहै (The advice that the app give is very good).
―App मेंजो prompt आतेहैंउसकीवजहसेमुझेज्यादासोचनानहीींपडता. App अपनामदमागलगाताहैं (The prompts
that come through the app ensure that I don’t need to think too much. The app uses its brain).
o A significant insight on the clinical impact of CDSS was that latent High-Risk Pregnancies
(HRP) were being uncovered.
One participant mentioned ―इससेहमेयेपताचऱजाताहैकि HRP हैकिनही‖ (With this we get to know
whether it is an HRP or not).
o Identification of high-risk pregnancy or a sick child ―Tab
ज्यादाजानकारीपूछताहै, हमेंपताचलजाताहैमकसको refer करनाहै, मकसकोनहीींकरना‖ (The tab asks for more
information and we get to know whom to refer and whom not to).
―हमारे काममेंबहुतआसानीहोगईहै. Referral cases पताचलजातेहै. बहुतसारे suggestions हमेममलतेहै. (Our job
becomes much easier. We get to identify referral cases and get a lot of suggestions).
Another participant mentioned ―हम HRP गभभवतीकोबताभीदेतेहैमकवजनकम हैं, खूनकमहै,
चलोहमआपकोमदखालातेहैं‖ (We tell HRP cases that your weight is less, or you have anemia, let me
get you checked).
o The Clinical Decision Support capabilities were lauded by the medical officers and nurses as
well.
One of the MOs mentioned ―इसमेकाफीऐसीचीजेंहैजोहमclinical practice मेंmiss करजातेहै (There are
quite a few things in this app that we usually miss in our clinical practice). A staff nurse said
287887/2021/DM&A
517 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
37
―जोचीजहमनहीींसोचपातेवोतुरीं तहमे suggest होजातीहै (There are things that we don’t think of, but that get
promptly suggested to us).
Another medical officer mentioned ―Paediatriciansकोसाराकु छयादरखनामुश्किलहोताहैं. Appमें age
के महसाबसेसाराकु छआजाताहै- treatment क्यादेनीहै, counseling क्याकरनीहै, diet advice येसाराकु छअ�ाहै
(It is difficult for Paediatricians to remember everything. The app provides everything based on the age - what
treatment to give, what counseling to provide, diet advice – all of this is good).
―Patient की history, symptoms, मकसे refer करनाहै CDSS सेपताचलजाताहै” (Patient’s history,
symptoms, whom to refer to – all of this we get to know from CDSS) said another MO.
The ability to automatically generate clinical summaries also seemed to resonate with the
participants. They appreciated the fact that a readily organized complete summary is available to
them at the end of interaction (for first responders) or when they receive a referral (staff nurses and
MOs)
o ―Summary मेंजोपूराrecord आजाताहै, उसमेसारीजानकारीहमेएकसाथममलजातीहै,
वोदेखके लगताहैकीवाकईमेंअ�ाहै. (The complete record comes as part of the summary and we get all the
information at once – looking at that makes one feel that this is really good).
o ―लेखाजोखासेफायदाहोताहै, कमसमयमेअपनाकरके लाभाथीकोभीसमझादेतेहै‖ (The clinical summary is beneficial,
we do our jobs in less time and counsel the beneficiaries as well).
o ―अपनेआप साराकु छ summarize होजाताहै. कु छही point हमे add on करनेपडतेहै, वनाभसाराकु छ summary
मेंआजाताहै (This automatically summarizes everything, and we need to add very few points, otherwise
everything comes in the summary).
Some of the participants, particularly at CHC, felt that it helped in an effective continuum of
care and possibility of faster care for the high risk / referred cases
o One medical officer mentioned ―पहऱेहमेसारीhistory ऱेनीपड़तीथी, अभीहमज्यादातरtreatment
part हीिरतेहै‖ (Earlier we had to take the entire history, but now we mostly focus on the
treatment part).
o One medical officer gave a clear description of the impact ―HRP load बढ़जायेगा,
परयहअ�ाहीहै. हमारे पासमफलहालANC case third trimester मेंआतेहैऔरउनकी screening
अगरASHA ANM करे तो HRP काशु�सेहीgrassroot identification होजाएगाऔरउनकाreferral and
management अ�े सेहोजाएगा, cases improve होजाएीं गे. (The HRP load has increased, however this
is good, because we currently get ANCs in the third trimester and if their screening is done by ASHAs
or ANMs, a possible HRP can be detected early with grassroot identification and their referral and
management will be better and cases will improve)
o Patient जबहमारे पासअचानकआताहैतोपताचलजाताहैमकहमनेउसेपहलेदेखाहैमकनही, या ASHA, ANM
देखचुकीहैमकनहीीं. (If a patient comes to us suddenly, we immediately get to know if we’ve seen the patient
earlier, or if the ASHA or ANM have seen the patient earlier) and we do not have to repeat the
whole history. “हमज़्यादाकामहोनेके बावजूदरेफ्रे डमरीज़कोदेखसकतेहै”We are able to help the referred
case appropriately despite our otherwise busy schedule,‖ said a staff nurse who immediately
thereafter had to rush to conduct a delivery.
There was also an overall sense of satisfaction with CDSSthat could be extrapolated from
comments such as ―हमेंमवश्वासहैइसपर‖ (We have trust in this), ―अभीहमइससेसींतुष्टहै‖ (We are now satisfied with
this) and ―App आगेभीचलेगातोज्ञानममलेगा‖ (If the app continues in the future, we will get knowledge).
Usability of the tool
As almost all the participants felt that the tool was useful to them in more than one ways, we wanted to know
why did it not reflect as emphatically in their choice for continued usage of the tool. What were the
hindrances and challenges which are not making it universally appealing?
287887/2021/DM&A
518 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
38
The biggest concern raised by several participants was the amount of time it takes for data
entry.
o One participant mentioned ―जबहमफीमडींगकरतेहैतोबहुतसीमदक्कतेआतीहै, जल्दीहोताभीनहीींहै.
एकएकमरीज़के फीमडींगमेंकरीबआधाघींटालगजाताहैकभीकभी‖(When we enter data there are a lot of
challenges, it does not happen quickly, it takes almost half-an-hour sometimes to enter data for one
patient).
On further discussions, it was clear that the longer time taken was due to several reasons,
many not directlydue to the current interaction, e.g.for the creation of the initial registration to
create a family record; due to the initial teething troubles and learning curve. This was also
reflected in the quantitative data as fall in incomplete journeys over time as discussed above (Fig 10).
o as one participant mentioned ―Register करके journey करनेमेंज्यादा time लगताहै, परअगली visit
मेंआसानीहोजातीहै‖ (It takes a lot more time for registration and then to complete a journey, but in the
next visit it becomes much easier).
o ―Starting हैइसमलएकु छissues लगरहेहै. करतेरहेंगेतोआसानहोजाएगा‖ (It is just starting which is why we
feel there are issues. If we keep doing it, it will get easier).
Another reason cited for more time required for the interaction using the tool was perhaps due to
inability to skip any part of detailed history taking while using app. quite unlike prevalent direct
interactions which are very pointed and limited like a firefighting approach.
o ―बहुतज्यादाचीज़ेपूछनीपड़तीहैं।कभीकभीतोबेनेमफशरीके हताहैक्यादीदीइतनासाराक्यूूँजाननाहैहमकोदेरहोरहीहै।(
We need to asktoo much in detail. Sometimes the beneficiaries tell us “Why do you need to ask so much.
We are getting late”).
However, when we asked which areas were unnecessary and can be trimmed, there were no
suggestions and often the group agreed that its lengthy but is useful.
o अ�ाहै, सबकामकीबातहैंबसहमअभीतकपूछतेहीनहीींथे।(It is good the way it is because it ispertinent, it is
just that we were not asking all this information till now).
It also become evident that the technology introduction was a disruptor in several ways. Currently
the interaction with beneficiaries is somewhat limited and the expectation of the beneficiary is to get
free vaccination or drugs and a quick turnaround. The time investment to create a detailed holistic
health record by the provider and the beneficiary has not been made in the past. The present tool
asks for more detailed and meaningful interaction to which both the stakeholders are not oriented.
o It is also challenging when multiple beneficiaries come together, as one participant
mentioned. ―Beneficiariesहमारेमहसाबसेनहीींआतेवोपूराझुींडबनाकेआतेहै. वहाींएकएककी journey
करनाimpossible होजाताहै. (Beneficiaries don’t come as per our schedule, they sometimes come as a
group, when completing a journey for each one of them is impossible).
o The beneficiaries are not willing / able to give time for the complete detailed history.
―लाभाथीसमयनहीींदेपातेहै, जल्दीमेंहोतेहै‖ (Beneficiaries are not able to give time, they are in a hurry).
o Moreover, it is currently not possible to see multiple patients together since the
app allows only one journey at a time. ―जबतकएक journey पूरीनाहोदू सरीशु�नहीींकरसकते
(Unless you complete one journey, you cannot start the next).
Using of a smart device for collecting data may be misinterpreted due to lack of awareness, lamented
a MO.
o ―Public ऐसीहैमकवहकहेंगे Doctor साहबmobile परखेलरहेहै, हमेनहीींदेखरहेहै” (The public here is such
that they would say the Doctor is playing on his mobile and not seeing us).
Another underlying concern which affected the choice of continuing adoption of the tool, particularly
among ASHAs, was the fear that it will increase their work load by increased identification of sicker
cases. Interestingly, in one of the FGDs, a very motivated and champion user wished to retract her
emphatic statement on the benefits of CDSS as it had made her co-workers unhappy for they felt she
was asking for a change that is going to increase their workload.
287887/2021/DM&A
519 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
39
o “औरज़्यादाबेनेमफशरीजकोसेंटरलेकरजानापड़ेगा‖ (We will need to accompany many more to the health
centers).
Thequantitativesurvey hadbrought about some concerns about the referrals being generated by the tool
as some of the recommendations for referrals were perceived to be unnecessary. However, the
participants were emphatic in stating across FGDs that they never had a situation where the app had
not suggested a referral when the workers felt it was required.The users agreed that it efficiently
identified all probable referral cases.
o ―कभीकभीapp refer करनेकोबोलताहैपरहमारीसमझमेrefer करनाज�रीनहीींहोता, परइसकाउल्टाकभीींनहीींहुआहै‖
(Sometimes the app asks us to refer but we don’t think referral is necessary, however the vice-versa has never
happened).
Discussion about possible facilitators, challenges and hindrances to adoption of CDSS tool brought
some more interesting insights
The reluctance to use the tool in part was because the users thought that it will be an additional
method and not a replacement of the physical mode as was clearly brought up at one of the
FGDs,involving 10 ASHA workers. When asked how many would like to continue using the app, 6
raised their hands. The four that did not, mentioned the following reasons for their reluctance: limited
time available and demanding work, lack of adequate incentives/payments, and the additional work
pressures during COVID-19. However, when asked that if the application were to replace the
handwritten ASHA register and other reporting requirements, all 10 ASHAs unanimously said they would
find it much easier to adopt. This was echoed by others too:
o One participant mentioned―डायरीवालाकामइसमेहोजाएतोबहुतअ�ाहै‖ (If the job of diary (daily
registers) can be done by this app, it will be very good).
o ―अभीदोदोकामहैं. अगर ASHA register कीजगहमसफभ येapp आजाएतोअ�ाहोगा,
क्योींमकबारबारभरनानहीपडेगा‖ (Currently we have to do two things. If the ASHA register gets replace by
this app, it will be good, and we won’t have to duplicate our work).
o ―OPD मेंdaily register भरनाकमहोजाएगा, हमारे पासएक handy app होजाएगाऔरबहुतआसानहोजाएगा‖
(If the need for filling out the OPD daily register gets reduced and if we have a handy app to do this, it
will become much easier).
However, there was a concern with a fully digital mode not being fully reliable
o One participant, who was an early adopter and had done a lot of cases, mentioned
―साराdigital होजाएये possible नहीींहै, थोड़ातो manual रहनाचामहए, अगरapp नाचलेतो” (It is not possible
that everything goes digital. Some of it should be manual – what if the app doesn’t work). (we will
have no backup).
The issue of incentives was also raised by some participants
o ―ज्यादासमयमनकालनापडताहै. इसकामके मलएहमेकोईअलगसे payment भीनहीींममलताहै(We need to take
out extra time for this and do not even get paid for the additional work).
But when we reminded them that the app is not adding any new work rather easing their work through in
a better and efficient method, it evoked mixed reactions from acceptance to need for better payment or
incentives as they felt that they were poorly paid.
Suggestions for improvement to improve usefulness and usability
Several participants had inputs on how to improve the app further. Although many of these suggestions were
predominantly around the app workflow, and not directly related to the CDSS, it was felt that these
suggestions, if incorporated, would increase the overall value, and consequently, the adoption of CDSS.
Internet connectivity emerged to be a critical bottleneck in adoption of CDSS and there was
a clear preference for an “offline” mode.
287887/2021/DM&A
520 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
40
o One participant mentioned ―सबसेबडी problem network कीहैं‖ (The biggest problem is that of the
network). Another suggested ―मबना internet केapp चलजाए to ज्यादाफायदाहोगा‖ (If this works without
the internet it will be more beneficial).
The ability to correct data entry errors was perceived to be a critical missing feature of the
product.
o ―गलतentry कोcorrect करपाए‖ (We should be able to correct wrong entries).
o ―मकसीलाभाथीका data feed करदेतेहैऔरवोहमेमकसीगलतीकीवजहसेchange
करनापडेतोहमनहीींकरपातेहैंऔरमफरसेहमे feeding करनीपड़तीहै‖ (If we make a mistake while feeding data
and need to change we’re not able to do this and have to start all over again).
o ―CDSS मे next औरbackका system होनाचामहए. कभीगलतीसेकु छ submit करे तोवो back
नहीींहोपाता‖(CDSS should have a system for going back and next. If we submit something by mistake, we
aren’t able to go back). नएमसरे सेकरनापड़ताहै. We have to restart the journey.
While the investigators agreed with the need but had concerns about the risk of data fudging by
enabling re-entries.
A recurring and consistent suggestion was that of capability to generate due lists, missed
appointment alerts, area or worker-wise list, an optimized workflow, and notifications to the
nurse or doctor on when the beneficiary is scheduled to be seen by them.
o ―Due List add होसके तोउससेकाफीसुमवधाहोजाएगी / ब�ोींके टीके की due list आनीचामहए‖ (If a due list can
be added, it will be very useful, a due list for pediatric vaccination should come).
o The need for a PNC due list was expressed clearly
―मडलीवरीहुईऔरहमप्रसवोत्तरमाहमेंफीमडींगकरनेचलेगएतोहमेंमदखाईनहीींदेता, तोहम CDS कै सेकरे(If a delivery
happens and the woman becomes postnatal, then we can’t seethe patient details in app so how can we do the
CDS?).
o “App ने indicate करनाचामहएकीकोइ visit छू टगई‖ (The app should indicate that a visit has been missed).
o ―गभभवतीममहला delivery के बादअपनेआप lactating mothers
कमलस्टमेंचलीजाएतोहमाराकामआसानहोजायेगा‖(After delivery if the mother gets added to the lactating mothers
list, it will become much easier).
o ―OPD में photo लेकरहमरातमे journey complete करतेहै. Time slot होऔर referral list
होतोअ�े सेकरपाएीं गे‖(We take photographs of the case record and complete journeys at night (because of the
rush in VHND). We will be able to do this better if there are time slots and referral lists).
o ―Referral मेंoption होनाचामहएकीहमकहा refer कररहेहै, मरीजकोकै सेभेजरहेहै, औरउनको alert करपाए. जैसे
CHC पेrefer करोतो CHC पेहीजारहाहो, मकस doctor कोrefer कररहेहै. जानकारीसबकोहोनीचामहएऔर tab
सबके पासहोनाचामहए‖ (There should be options in referral – where are we referring, how are we sending the
patient, and alerts. E.g. if we are referring to the CHC, the patient should go to a particular CHC and to a
specific doctor. Everyone should have the information, and all should have the tab).
o ―VHNDमे patient Register करकेjourney करनामुश्किलहै. हमेपतानहीींचलतामकASHAनेVHND
के मलएpatient Register मकयाकीनहीीं. कभीकबारdouble registration भीहोजाताहै‖ (It is difficult to register
patients during busy VHNDs. We don’t get to know whether the ASHA has registered the patient for
VHND. Sometimes there is duplicate registration).
o ―Journey next day करनेकीसुमवधाहोनीचामहए, कु छमदक्कतहोतोलाभाथीक specially बुलाकरadvice देपाए‖
(There should be a facility to complete a journey on the next day. If there is a problem, we should be able to
specially call the beneficiary and give them advice).
Medical Officers suggested some improvements in clinical content.
o One MO mentioned ―इसमें NCDs add होनीचामहए‖ (This should also cover Non communicable
diseases)
o Another MO suggested ―इसमेdifferential diagnosis का hint होनाचामहए, मजससेहमारीpersonal
knowledge में improvement होसके” (This should have hints on differential diagnosis, so that there is
improvement in our personal knowledge).
o “थोडी language clear होनीचामहए. एक answer के दोअथभनामनकले” (The language should be clear. One answer
should not have two meanings).
There were concerns about the multiplicity of mobile apps, particularly the ANMs.. One
respondent mentioned that there were too many apps being introduced. The Government is also
287887/2021/DM&A
521 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
41
releasing the ANMOL app which they are required to use. If all of them get integrated into one
simple app, it will be much easier to use.
7. Discussion
This feasibility study to assess the usefulness, usabilityand adherence to STGs, with the use of CDSS in
Indian public healthcare settings generated significant insights into the pre-study rationale, beyond
conclusively proving the research hypotheses.
Transformation of clinical guidelines to a digital interactive smart tool
Feasibility of transformation of local guidelines and developing smart clinical decision support tools for use
by health functionaries in the community and primary care settings was firmly established. Even though this
pilot proof of concept study had worked only with few maternal and pediatric (under five yrs. age)conditions
considered important from public health viewpoint, yet it demonstrated the ability to convert these into a
reliable digital tool with complete fidelity to the written guidelines across various selected conditions and even
with situation where more than one of these co-existed. The users had a choice to use the tool either in Hindi
or English language.
Empowerment of Front-Line Healthcare Workers
There wasvast appreciation of the ability of the tool to usefully add value to interactions with the
beneficiariesby the users.As was evident from the surveys and FGDs, the CDSS intervention empowered
FHWs at every stage, with the augmentation of knowledge that came sequentially at the right point
in the flow in each screening journey, so it acted as a fantastic job aide for everyone. This came out
strongly from all the three cadres of the study participants. While ASHAs and ANMs remembered very less
of their pre-study guidelines training sessions, the app reduced such kind of memory attrition, because the
tool takes one through the whole process of evaluation and brings up all the relevant questions required for
an individual patient, and the FHWs do not have to struggle to find what guidelines apply. There was also a
strong sense of empowerment since the study extended the FHWs‟ horizon into some areas that they
had previously never worked on, like developmental milestones, which they‘d never used in the past
because the recently introduced MCP card was not a part of their routine.
Broadly, the benefits accrued are described here.Firstly, the knowledge of FHWs was augmented through
CDSS, based on reliable and extensive information gleaned from selected guidelines. Relevant advice within
the CDSS app was generated at the right time and strategically placed to be invoked when it was required at
the point-of-care. Normally they would have to refer to a guideline and look up static information, but the
CDSS app is designed in such a way that relevant clinical information came as part of the flow, at the right
place and at the right time, empowering FHWs with knowledge in context of the workflow. Secondly the
attitude of FHWs was also positively impacted through better structured advice. They felt more confident
in their interactions and appreciated the structured and detailed action-oriented advice relevant to the
caseproduced by the tool.Thirdly, the workers also appreciated that the tool was more comprehensiveand
reliableas it never missed a case which required referral. The structured summaries which were available
at the end of an interaction made the functioning efficient and complete. This critical point of impact was
made by several participants, particularly the MOs as they highlightedthe ability of the app to empower
ASHA‟s and ANM‟s to systematically screen, identify and timely refer high-risk cases particularly
HRPs in early pregnancy rather than in third trimester when it is too late. The MOs expressed a clear desire
to augment the CDSS app with additional elements like differential diagnosis and suggested
inclusion of conditions like non-communicable diseases (NCDs), indicating need for feature
augmentation for improved acceptability.
Task Shifting and provision of seamless continuum of care
Although the actual process of effective task shifting is a gradual process, there was a successful
demonstration of the potential of task shifting with the CDSS intervention as part of this study. Due to
287887/2021/DM&A
522 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
42
the thoroughness of the app in capturing an entire visit in the form of a lucid summary, by the time the
beneficiary reached the next level of care, a significant part of necessary clinical information was already
available, and it reduced duplication of effort that occurs each time a frontline worker interacts with a patient.
The app can also serve as a two-way communication tool, between ASHA and ANM for seeking
opinion and advice by sharing case summary and between ANM and MO, without the ANM
needing to accompany the patient to higher center. The ANM can send details that can be accessed by
the MO and initiate management under the supervision of the MO. This provides a unique opportunity of
continuum of care for an individual as long asthey are within the system. This also provides an opportunity
to seamlessly provide escalation and de-escalation of healthcare-related activities to any beneficiary.
As there was clear documentation of the interaction and the MO has the opportunity to verify the inputs
from ASHA/ ANM on a particular case, it provides an opportunity for on-the-job support and training to
these functionaries through feedback. The opportunities to analyze which healthcare workers need to be
trained on better health screening, based on information about practices available within the system, can
accommodate capacity-building efforts towards effective task shifting. The timely and contextual
cognitive guidance willmake theASHA or ANM more confident in taking on higher responsibilities with time.
Through the CDSS App, information capture has partially moved to the lower cadre and treatment has
moved to the upper cadre in the field, whereby medical officers just verify the history and assessments,
reconfirm and add on to the treatment, if required, resulting in an optimized workflow and conducive
grounds for task-shifting. In addition, this also provides credible information for supportive supervision
across the cadre.
The clinical impact of such task-shifting could be significant. Clear communication between ANM and MOs
can optimize referrals, and early detection of HRPs, when the ANM is able to send a list of HRP referrals
and the staff nurses as well as MOs at the centers exactly know which patients are expected and whom to
prioritize. This could also reduce waiting times for HRPs who would receive treatment and intervention
earlier. Further, follow-up care could also be shared with the MO in a two-way communication
model, allowing patients to be sent home on supervised domiciliary treatmentwith effective follow-up care by
ANM and ASHA. This, in some ways, is step-down care in the reverse continuum, which is also one aspect
of task shifting, economical and a patient and facility-friendly output. Some eligible high-risk patients can get
care at home without disrupting their family life, income, or affecting caregivers, by seeking care closer to
home and improving its acceptability. It will also allow fast turnover of inpatients in the higher centers like
CHC. Task shifting is not merely about a complex task being performed at a lower hierarchy, but it is also
sharing of tasks, which is extremely important from a public health point-of-view, especially in terms of
breaking the silos, that most systems are currently working in,thus making the care continuum more efficient.
Acceptability of CDSS
Usability and usefulness of CDSS were two primary outcome measures that inherently impact its acceptability
by FHWs. Some manifest benefits that accrued with CDSS included the optimization of the continuum of
care through better two-way communication between the ASHA, ANM, Staff Nurses and Medical Officers
based on the automatically generated clinical summaries obviating the need for personally accompanying
patients. Also, a longitudinal record grounded in health screening and clinical information ensures
continuity of care even if physical records like the Mother-Child Protection (MCP) card are misplaced.
Acritical insight was that workflow improvements in the operational part of the App will have a bearing
on the acceptability of CDSS since the operational and clinical elements of health screening go hand in
hand.The ability to integrate optimized workflows with CDSS recommendations is a powerful combination,
allowing for due-lists, alerts, and reminders along with evidence-based screening and recommendations. This
is do-able but was beyond the scope of the present proof of concept study which largely focussed on
establishing feasibility of the idea of adoption and conversion of the guidelines to a computer interpretable
format and creating a decision support system customised to the country guidelines.
Data Provenance
Arezzo
® demonstrated the ability to generate high-quality clinical data sets during usage as part of
the FHWs workflows during the study and this holds a lot of potential for further analysis and use in the
287887/2021/DM&A
523 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
43
future. This operational and clinical data can lend itself to post-hoc analytics leading to a secondary gain in
terms of an increase in district-level data about care delivery, referrals, and outcomes. It is also critical
to realize that while electronic data does get captured as part of the workflow, the eventual analytics
processes need to be thought out at the outset, specifically in terms of what reports, dashboards and
views are required, how the data is indexed and standardized. The granularity of this data can be from the
district-level to the PHC-level and further to the individual FHW-level, signifying a major potential for
supervision and real-time monitoring. It must be supported with a simplified information retrieval process
which is navigable evenby non-technical functionaries. The eventual goal would be to ensure that the desired
analytic views are available at the click-of-a-button with the ability to drill down to the individual FHW-level,
which is critical in terms of root cause analysis in terms of contributory factors such as deficiencies in
infrastructure, training and guide actions required. This will enable early identification of weak spots in
the system and weak links to direct focussed interventions to strengthen them instead of generalized
interventions for all. This couldresult in effective and real time course correction and enable the system to
reach predetermined goals efficiently without waiting for time and manpower consuming nationwide surveys
every 5-10 years. Also, the possibility of district-level dashboards linking outcome measures to region-
specific practices and training needs for FHWs is a significant public health opportunity. In fact, if used at
scale, it holds the potential to partially automate NFHS-like national surveys.
8. Lessons learnt and lateral insights for future studies
Lack of Harmonized Standard Treatment Guidelines
The biggest issue that the research team faced in adaptation of STGs was the lack of harmonized and
updated guidelines for most conditions. There were some variances observed between guidelines. Thus,
e.g., what the ASHA Module recommends was at variance from what the SAANS (Social Awareness and
Actions to Neutralize Pneumonia Successfully) program contained. The research team had to reach out to
several functionaries to get a sense of the expected updates to guideline since they were cautious of
introducing any element that was not part of existing guidelines and that could confuse health workers. A
solution to digitize guidelines does partially address this issue, primarily by identifying the underlying
fragmentation in order to institute corrective action. Another challenge faced was due to vertical health
programs that are focused on one problem, but when it comes to an individual patient there usually are
overlapping symptoms, and this can create a lot of confusion, especially with more than one terms describing
it across different guidelines. Thus, the need for harmonization of guidelines is also at the level of
guideline-specific-language. The PIs and ERG had to spend efforts tomerge multiple guidelines and
help the developingteam to create CDSS solution with the ability to render them seamlesslyto the FHWs,
and thus the Arezzo
® technology could facilitate the continuity of care required with comorbidities,
complications, and disease clusters.
Another critical issue faced by the research team was the presence of rather outdated guidelines. Maternal
guidelines from GOI have not been updated for quite some time now, and thus there were mismatches
between practices at the ground-level and published guidelines, especially for ASHAs and ANMs.
There were also some published guidelines that had not been implemented yet such as Anemia MuktBharat,
GDM guidelines, and the research team had to join those pieces together. While health is primarily a state
subject, there are vertical National Health Programs financed by the Central Government. But there are
variations at the State-level implementation of these programmes as the State can always adapt these
programmesby whatever they think is relevant to their own region.Also, some elements that are there in the
guidelines may not exist in the periphery, especially due to lack of resources. However, the starting point of
such a digital solution should be a common minimum agenda that is acceptable across different state
level health functionaries. It is important that the final product is not devised in purely top-down approach
and includes local adaptations for a confusion-less transition to digital mode. Importantly, theprocess of
digitization makes STGs more dynamic, improves the ability to harmonize various guidelines,
leading to faster dissemination and therefore, may be more efficient way for implementing changes
/ modifications to guidelines. Further the ability to capture practices in different areas enables uncovering
of gaps that can be addressed in trainings or updates, thus improving the process of implementation of
guidelines.
287887/2021/DM&A
524 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
44
As for the adaptation of STGs to CIGs,the process of content transformation to digitize guidelines was
quite challenging. Firstly, a change in the mode of delivery of guidelines into a format where the FHW
would screen beneficiaries with initial questions and then leading questions based on previous responses
required a thorough deconstruction followed by reconstruction of the STGs to align it in a conducive format.
Further it took time for the research team to become familiar with the content transformation processand for
the technology team to become more aware of locally relevant clinical practices. Furthermore, the need to
incorporate increasing levels of sophistication in the upstream hierarchy of the healthcare workers required a
cadre-wise distinction of various parts of STGs that did not always specify who had to execute what part of
the guideline and how one could cater to variable patient cohorts based on their socio-economic status and
level of education. This further complicated the scope of the content transformation process. Also, the level
of manual effort involved in making small changes to the CIGs was something that took time. However, the
delay in starting phase 2 due to COVID allowed a prolonged Phase 1 to the investigators to fine tune the
conversionprocess. In the initial phases, a small change would shake the whole journey so incorporating
changes in the app was a challenge and time-consuming task.
Local-language translations posed a significant challenge in this study. There was significant effort put
in by the research team for Hindi translations and the process of using spreadsheets to communicate these
translations to the technology team based in London (not familiar with Hindi) was not optimal leading to
several iterations and rework.Although the local-language at the study site was Hindi and most of the research
team was comfortable with Hindibut further changes had to be incorporated later in the implementation
phase based on the user feedback. This largely resulted from differences from the acceptable or
understandable vocabulary /locally used hindi words. Future scale-up efforts into other local languages will
require fool-proof translation processes with involvement of few well-versed local health functionaries early
on.
Lack of protected time or incentive for study participants, COVID pandemic related disruptions
Time and timing (occurrence of pandemic) were the biggest limiting factorsfor the study. The teams
were working across continents and the time differences impacted coordinationand added to delays despite all
efforts by the teams to find a time slot on a regular basis. Notwithstanding extended study period due to the
COVID-19 pandemic, local restrictions and reorientation of the health care entailed that the opportunities
were still limited for effectively evaluating workflow optimization. As ASHAs are notfull-time health
functionaries, absences due to local festivals, social reasons and health or confinement also impacted the
workflow.
Many homebased activities were suspended due to COVID hence related guidelines could not be evaluated in
the context of continuum of care visit by visit. It affected evaluation of interventions during follow-up visits
and the impact of the Home-Based New-born Care program. The study team could not stay in the field for
long time for non-participant observations of the interactions. Shadowing of interaction with the beneficiary
which were planned real time was not possible even virtually due to network issues in the field. The team had
to resort to deferred virtual assessment and facilitation, usually in the afternoons or evenings after the
workers were back to their stations.
With the raging pandemic that required the FHWs to put in additional effort for COVID-19 screening and
vaccination, there was a significant increase in their workload and lack of a regular availability of the
workforce for the study purpose.There was also an underlying concern amongst the FHWs that the app is
likely to increase their workload without an increase in their remuneration. This feeling came in due to
required partial duplication of effort since the existing manual workflows continued and the FHWs had to
make data entries into the app for beneficiaries that met the inclusion criteria only for the purpose of the
study. This led to a slow uptake by the workforce. The FHWs did not fully understand that this was in a
study-mode and that the workload would decrease after the study, either by replacing the manual registers
with the app, or by going back to status-quo, which is the reason why some of the FHWs were hesitant and
not fully forthcoming on their feedback during the FGDs. The critical support by project execution partners
Piramal (PSMRI) in the field under direct supervision by NITI Aayog was greatly helpful in execution of this
project by convincing, motivating and monitoring the FHW participating in the study.
287887/2021/DM&A
525 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
45
Nascent technology implementation in geographies with infrastructural challenges
The project was testing implementation of a customised technical solution in a nascent area in an aspirational
remote district of the country. In addition to the customisation and adaptation it required a local vehicle for
frontline access. In the absence of any individual electronic health records, it required a local digital vehicle to
capture the patient or beneficiary related data. The local server then had to connect with the customised
CDSS tool in an anonymised way. This system architecture created many challenges as it required a well
synchronised functioning. As the Amrit app on which the CDSS tool (Arezzo) had to ride was also new and
in its agile evolutionary phase, there were time lags in identifying problems and providing solutions.
This first prototype developed under the project lacks many enhanced features in its user interface as
CDSS had been riding on a new app environment. The prototype app also did not have an edit
capability or the ability to go back and forth between screens, which resulted in the need for restarting a
journey if a data entry error was made. This requirement was unanimously voiced by most participants during
the survey as well as the FGDs and is a critical requirement for future releases of the CDSS app.
In particular, the need for due lists and appointment-scheduling was a significant requirement stated by the
FHWs. However, there was also some hesitation in expressing this feedback clearly, since the addition of
such features would increase the transparency in the system leading to a higher level of accountability on the
part of ASHAs and ANMs. Get data functionality essential for continuum of care from ASHA to ANM to
MO could be made available towards the end of study with limited time for the HCWs to assess its
usefulness.
There was also a learning curve with the app. The first visit involves registration of the beneficiary and
takes time to enter demographic and social information before a CDSS journey can be initiated. Thus several
participants reported usability issues, in terms of time, however, it was also evident that during subsequent
periods the workflow was more efficient and progressively the data entry requirements reduced and the
CDSS aspects of the app became more prominent, the journey itself took much less time. Moreover, the
CDSS app ensures detailed history which wasn‘t elicited earlier and the FHWs weren‘t accustomed to this,
but over the study period the benefits of this initial history became apparent, and several users gave positive
feedback about its usefulness, despite some issues with usability in the short term.
Network connectivity was a significant bottleneck, and it was evident that the clear preference is for an
app that can work in the offline mode. Technologically, it is possible to create an asynchronous app that has a
device-level memory and that can further be synchronized once network is available, however this kind of
app architecture was not available for the study and connectivity did impinge on the usability of the app
significantly.
Hesitancy and unfamiliarity withlikely benefits with new technology among beneficiaries
There were quite a few challenges with the CDSS workflow when beneficiaries presented as a group,
like in the case of VHNDs. While the expectation of beneficiaries is to be done within 2 or 3 minutes, the app
would require significantly more time, sometimes up to 30 minutes, and the beneficiaries were not willing to
wait for so long, due to their domestic and work-related priorities. Further, without a prior experience or clear
understanding of the benefit that such systems will provide to the beneficiary there was also no clear
trade-off that was apparent to the beneficiaries who were part of the study. The pertinent issue that emerges
here is what incentives do beneficiaries have to participate in a health system that is enhanced with
knowledge-based technology and addressing this would be a major implementation challenge in the future.
This in many ways requires a cultural change in the community and their expectations from the healthcare
delivery system. This comes with experience of the benefits and its word-of-mouth propagation within the
community. The value of preventive and promotive care in the minds of most people, including
educated people, is far less than the curativeservices. This is essentially the area that requires a cultural
shift in the community as well as the workers.
Study design and plan limitations
A major limitation of the study was that it this initial POC prototype was restricted to 10 conditions in
maternal and child health. This meant it was not possible to track all potential comorbidities and disease
287887/2021/DM&A
526 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
46
clusters. Several beneficiaries did present with other common conditions or were of a different age group and
the study design did not accommodate all of those due to its scope and inclusion criteria. An initial study had
to avoid an uncontrolled scope that would have impinged on the primary outcome measures of usability,
usefulness and adherence to STGs. However, it would be ideal to overcome this limitation in future study
designs to assess the impact of a wide coverage of conditions like communicable and non-communicable
diseases. It may be useful to do a pre study survey in the facilities to be included and list out all possible
common conditions with which patients under consideration present and include most of those so that the
HCWs use the app for all or most and not selectively.
There was also a limitation in terms of the geographic coverage of the study sites. Only a limited
number of SCs, PHCs, CHCs and their constituent health workers were a part of this study. This impacted
the follow-up of patients and referrals at the CHC, limiting the process of tracking referred patients.
Future implementation studies which will require some further degree and detail of customisation (e.g. being
mindful of local language, cultureand practices) shall need to be planned in a more participative manner
rather than a top down approach. The team may need to shadow-learn working of the end users and take
specific inputs from them before and during the study to fine tune the final product as per their working and
job needs. The planning period has to be fairly long and widespread to capture a detailed picture. The current
study could have resulted in a better product if we could have done all the planned interactions which got
impacted due to COVID. Learning from the improvement philosophy of Japanese industry - ―Kaizen‖ may
be pertinent here. It has three important components called 3G; which involves having the managers go to
the actual floor (gemba); looking at the actual products involved (gembutsu) and gathering as many facts about
the situation as possible (genjitsu) to plan improvements.
It is also important to consider all the preparatory elements required for the study. The need for a stable
app, local language translations, technical integration to be completed prior to the commencement of the
study was a key learning lesson from this study. Further the study site selection, transition period for training,
absorption of technology, as precursors of the study would ensure a higher level of preparedness before
initiating the actual implementation process and ensuring that implementation challenges do not
obscure the actual benefits of the study. The preoccupation of the HCWs with their daily activities
interspersed with focused activities left them with limited time to focus on learning this new technology.
Familiarizing all HCWs with electronic data management of all beneficiaries would be a welcome first step for
sensitizing them to this concept. Use of CDSS could then follow.
Our study was designed as a qualitative feasibility study and did not directly address the impact of
CDSS on health outcomes. After having conclusively ascertained the feasibility of implementing CDSS in
public healthcare settings, through primary outcome measures of usability, usefulness, and adherence to
STGs, future research efforts into CDSS need to be directed towards impact assessment, aimed at the
direct effect of a CDSS intervention on health outcomes.Such kind of an impact assessment would
require a different study design and would need to be conducted over a longer duration since the effect of
CDSS on health outcomes is a gradual process and it takes time for such interventions to start becoming
visible in direct health outcome measures related to clinical effectiveness, morbidity and mortality. The
indicators used for the study were, firstly, on the perception of participants and, secondly, on adherence to
guidelines. An impact assessment study would entail verification of compliance on the ground, either through
secondary surveillance systems or direct supervisory monitoring, aspects that were out of the scope of this
study.
While a Cluster RCT with and without the intervention of CDSS could be a valid study design, this
methodology is fraught with the risk of uncontrollable confounding variables between two study sites
– in terms of locally relevant practices, language, dialect, infrastructure, healthcare administration and similar
factors.
A before and after study design could partially offset these confounders but this need be calibrated
with the right mix of study participants, clinical conditions, and an appropriate study duration.
Ideally a study design that has an initial ‗before‘ phase to evaluate current knowledge, attitude, practices and
health indicators needs to be sequenced with an ‗after‘ phase that has a considerable ‗run-in‖ phase
287887/2021/DM&A
527 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
47
toaccommodate the time it takes for any intervention to have an actionable impact on health outcome
measures. This duration needs to be ascertained by statistical extrapolation of existing health indicators and
ongoing health programs but is not likely to be lesser than 24-36 months.
Further the choice of primary outcome measures is critical. While the current study design included
quantitative STG adherence measures, these were based on compliance of FHWs to recommendations
contained within STGs. For a through impact assessment study the indicators would be more
proximal to the actual health status of beneficiaries and would be a combination of process and
outcome indicators. E.g. Anemia would include process indicators relating to the prescription of IFA, Anti-
helminthics, medication adherence at the community level, and outcome measures relating to the Hb trends
at the population level, and the impact of interventions on anemia over a period of time, since the
supplements would take at least 12 weeks to have a measurable impact on anemia at the population level as
well as in special groups like post-partum patients. Also, it would require that the outcome measures be cross-
functional since simple interventions in combating anemia would result in remarkable improvements in other
health indicators too.
The future studies certainly also need to take into account the technical issues related to functioning of the
app and the intelligent controller ecosystem (software, server, internet access, rapidity of response, etc.).
Expanding from learnings from the study, with inputs from the multi-disciplinary steering group, the way
ahead will need to use enhancements of the app as well as beyond it. The possible preparatory steps will need
development of, action oriented implementable guidelines to cover the whole gamut of primary health care
services and convert them to computer interpretable formso that the app replaces the paper registers in the
implementation study area completely. The tool should have capacity to be the first care step for patients to
get advice.
While that advancement happens, testing of the app with a large base of end-users may be useful to find,
execute and fine-tune some important technical solutions required to remove the stumbling blocks, e.g.:
a. Solutions to decrease latency and ensuring almost no denial of service from the server when multiple
users are using the app at the same time, due to which the requests may not be serviced by the app.
b. Solutions to decrease impact of lag due to slow internet connectivity in fringe areas by:
reworking out app with whole corpus of questions, prescriptions and other details being
locally present in the handset itself or have some other local caching mechanism available
over the site. It will also help to prevent fast drainage of battery owing to much network
traffic going back and forth.
using local edge client servers for making sure that static components of the content are
delivered from local edge servers, and dynamic components can be fetched from the cloud
periodically to ensure there is low latency in content fetching.
Possibly, an easy to use interface can also be developed to assist self-guidance of the patients, based on their
symptoms (like tele-medicine). Technology aided customized solutions have the potential for improving
access to appropriate care through intelligent decentralized guidance. The use of telemedicine has just started
during the current pandemic and definitely looks promising to continue as an additional tool for expanding
the reach of health care.
Beyond the context of the present study, we do realize that there are immense possibilities for use of
technological solutions to fill gaps and strengthen health care. Partners from technological, medical and field
implementation ecosystems within India should work together to create solutions using artificial intelligence
and machine learning. We need to test how far the conversational AI can be developed to improve the access
for those who find it difficult to read and respond.
Intelligent solutions could be devised to harness the information collected through community users to
identify early trends of a disease outbreak or to assess the health of the community from the nutritional data,
illness records, etc. The use of digital technology could not only help task sharing and shifting across the
various level of health functionaries but also between the man and machine, thus filling up some of the gaps
due to low numbers of trained doctors in the periphery.
287887/2021/DM&A
528 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
48
9. Conclusion
The present study conclusively demonstrated the feasibility to transform country guidelines and create
advance data analytics to develop acustomized clinical decision tool. It required the local experts to
harmonize and develop the guidelines to assist their metamorphosis to computer interpretable format. The
expertise of the creative partners at Elsevier ensured integration of “Arezzo®” - an active Declarative AI
clinical decision support system toan indigenous local app.Theadolopementprocess had utilized all the
elements viz. adoption, adaptation as well as development to create the final prototype.
We also established that the integrated CDSSis usable and useful to health workers in primary and
secondary healthcare settings and is preferred by over three fourths of the users for continued
adoption. Provision of Arezzo® customized for Indian guidelines-based recommendations empowered the
frontline worker as it acted as an able job aide with resultant improved recognition of high-risk cases in the
community. It showed added potential for efficient task share and task shift of care across the health care
functionaries from community to FRUs and vice versa. The study results are convincing enough to support
development of next version tools with further enhancements.
287887/2021/DM&A
529 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
49
10. References
1. World Health Organization. WHO Country Cooperation Strategy At a Glance: India. 2018.
2. World Health Organization. Maternal mortality - key facts. https://www.who.int/news-room/fact-
sheets/detail/maternal-mortality. Published 2018.
3. NITI Aayog. Maternal Mortality Ratio (MMR). https://niti.gov.in/content/maternal-mortality-ratio-mmr-
100000-live-births. Published 2018.
4. Lawn JE, Blencowe H, Kinney M V, Bianchi F, Graham WJ. Evidence to inform the future for maternal and
New-born health. Best Pract Res Clin Obstet Gynaecol. 2016;36:169-183.
5. Dhirar N, Dudeja S, Khandekar J, Bachani D. Childhood Morbidity and Mortality in India—Analysis of
National Family Health Survey 4 (NFHS-4) Findings. Indian Pediatr. 2018;55(4):335-338.
6. Soni M, Agrawal S, Soni P, Mehra H. Causes of Maternal Mortality: Our Scenario. J South Asian Fed Obstet
Gynaecol. 2013;5(3):96-98.
7. Organization WH. WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience. World Health
Organization; 2016.
8. Ministry of Health and Family Welfare G of I. Guidelines for Antenatal Care and Skilled Attendance at Birth by
ANMs, LHVs, SNs. https://www.nhp.gov.in/sites/default/files/anm_guidelines.pdf. Published 2010.
9. Ministry of Health and Family Welfare G of I. Janani Suraksha Yojna.
http://pib.nic.in/newsite/PrintRelease.aspx?relid=123992. Published 2015.
10. Ministry of Health and Family Welfare G of I. Janani Shishu Suraksha Karyakram : Saving Mothers and New-
borns.
11. Singh PK, Rai RK, Alagarajan M, Singh L. Determinants of maternity care services utilization among married
adolescents in rural India. PLoS One. 2012;7(2):e31666.
12. ICF II for PS (IIPS) and. National Family Health Survey (NFHS-4), 2015–16: India. 2017.
13. Mavalankar D, Vora KS. The Changing Role of Auxiliary Nurse Midwife (ANM) in India: Implications for Maternal and
Child Health (MCH). Indian Institute of Management; 2008.
14. Ministry of Health and Family Welfare G of I. Rural Health Statistics. https://nrhm-
mis.nic.in/Pages/RHS2018.aspx?RootFolder=%2FRURAL HEALTH STATISTICS%2F%28A%29 RHS -
2018andFolderCTID=0x01200057278FD1EC909F429B03E86C7A7C3F31andView=%7B09DDD7F4-80D0-
42E3-8969-2307C0D97DDB%7D. Published 2018.
15. World Bank Data. Rural population (% of total population).
https://data.worldbank.org/indicator/SP.RUR.TOTL.ZS?end=2018andlocations=INandstart=2018andview=b
ar. Published 2018.
16. Chavda P, Misra S. Evaluation of input and process components of quality of child health services provided at
24 × 7 primary health centers of a district in Central Gujarat. J Family Med Prim Care. 2015;4(3):352-8.
17. G Sugunadevi. Quality of antenatal care services at subcentres: an infrastructure, process and outcome evaluation
in a district in Tamil Nadu. International Journal Of Community Medicine And Public Health, 2017:4(11):4071-
4077.
18. Praveen D, Patel A, McMahon S, et al. A multifaceted strategy using mobile technology to assist rural primary
healthcare doctors and frontline health workers in cardiovascular disease risk management: protocol for the
SMARTHealth India cluster randomised controlled trial. Implement Sci. 2013;8(1):137.
19. Callaghan M, Ford N, Schneider H. A systematic review of task-shifting for HIV treatment and care in Africa.
Hum Resour Health. 2010;8(1):8.
20. Joshi R, Chow CK, Raju PK, et al. The rural Andhra Pradesh cardiovascular prevention study (RAPCAPS): a
cluster randomized trial. 2012.
287887/2021/DM&A
530 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
50
11. Annexures
Annexure 1: Arezzo® Validation
The regulatory authority in the UK (MHRA) has advised that Arezzo does not need to be registered as a
medical device currently. This is because: Arezzo provides explicit recommendations with clear rationales (i.e.
is not a black box); the clinical user can review all data that has been used by Arezzo; and the user can access
the primary source content at any time. External Legal Counsel has confirmed the same findings for the FDA
and the European regulatory authority.
The original research work for Arezzo® was carried out by Imperial Cancer Research Fund (ICRF) in the
UK. The work continued under Cancer Research UK, after the merger of ICRF. ICRF set up the Advanced
Computational Laboratory with the primary research objective of ensuring existing cancer treatments of
known efficacy are used to maximum benefit, whilst new treatments are being developed. The research and
early prototype work are set out in the book ‗Safe and Sound: Artificial Intelligence in Hazardous
Applications‘, written by Professor Fox and Dr Das (ISBN: 978-0-262-06211-4).
Following delivery of the first Arezzo prototype, a series of research studies were carried out. These studies
were the equivalent of phase 1/2 trials, seeking to understand what the potential benefits might be within the
research laboratory:
CAPSULE Study
• Research objective: determine potential impact of Arezzo on prescribing by GPs
• Research design: Decision support vs unaided choice; 42 UK GPs, who made choices from a simple
alphabetic list (control condition) or a computer-generated shortlist with a patient-specific rationale
(arguments for and against each option on the shortlist).
• Outcome/s: CAPSULE demonstrated a 70% increase in GP decisions that matched those of an expert
panel; a 50% increase in selection of a cheaper but equally effective drug, and decisions were made 15%
more quickly.
• Publication: ‗CAPSULE: Routine prescribing in general practice‘. Walton et al. British Medical Journal
1997, 315:791
CADMIUM Study
• Research objective: determine potential impact of Arezzo on detection of breast cancers during
mammography screening.
• Research design: Decision support vs unaided interpretation; 4 radiographers made decisions based on
mammogram images with and without CDS interpretation.
• Outcome/s: Radiographers trained to interpret mammograms performed better when using the advice
provided by CADMIUM: correct assessments of abnormalities were significantly increased and incorrect
judgements were significantly reduced.
• Publication: ‗CADMIUM: Detecting abnormalities in breast cancer screening‘. Medical Image Analysis,
Taylor et al 1999, 3 (4), 321-337.
RAGS Study
• Research objective: determine potential effect of Arezzo CDS on detection of familial cancer risk.
• Research design: Argumentation vs statistical vs paper and pencil; 36 GPs managed 18 simulated patients:
6 with a computerised decision support system (RAGS), 6 with an established pedigree drawing program
designed for clinical geneticists (Cyrillic), and 6 with pen and paper.
287887/2021/DM&A
531 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
51
• Outcome/s: RAGs resulted in significantly more appropriate management decisions than either Cyrillic
or pen and paper. RAGs also resulted in significantly more accurate pedigrees than the other methods. 33
doctors (92%) preferred using RAGs overall.
• Publication: ‗RAGS: Computer support for interpreting family histories. Emery et al British Medical
Journal, 2000, 321; 28-32.
LISA Study
• Research objective: improved accuracy of prescribing.
• Research design: Decision support vs unaided care; A web-based decision-support system was designed
to facilitate access to full blood count information across geographical locations and to assist with dosage
for maintenance chemotherapy. 36 clinicians with varied experience decided on oral chemotherapy doses
for 8 simulated cases: 4 with CDS; 4 without.
• Outcome/s: Significantly reduced the number of erroneous prescriptions (0/144 with CDS vs. 54/144
without; P <0.0001).
• Publication: ‗LISA: Chemotherapy decisions in acute lymphoblastic leukaemia‘. Bury et al British Journal
of Haematology, 2005, 129, 746-754.
• These studies suggested that Arezzo could provide benefit, which led to more rigorous randomized
controlled trials:
HAVANNA Study
• Research objective: reduction in HIV viral load.
• Research design: Cluster-randomised, multi-centre international controlled trial; HIV/AIDS patients
underwent viral genotyping; HIV specialists randomized to use CDS or clinical judgement to prescribe
anti-retroviral therapies.
• Outcome/s: 30% reduction in viral load in patients treated with Arezzo versus no CDS.
• Publication: C Tural et al. The Havana Trial. AIDS. 2002; 16:209-218.
FASTEST Study
• Research objective: reduction in all-cause mortality and vascular events; adherence to guidelines.
• Research design: Multi-centre, single-blind, cluster randomized, controlled trial (29 clinics) comparing
electronic decision support guided management with usual care for TIA/stroke.
• Outcome/s: More intervention patients received guideline-adherent care (131/172; 76.2%) than control
patients (49/119; 41.2%). 90-day stroke or TIA occurrence was lower in the intervention group (2.3%)
than the control group (8.5%). User feedback was positive.
• Publication: ‗Diagnosis and management of patients with acute stroke‘. Ranta et al Neurology 2015 14;
84(15), 1545-51.
Several non-regulatory validation studies have been carried out with the national implementations of
Arezzo®:
• Early Referrals Application (ERA) – implemented UK Department of Health guidelines for improving
early detection of suspected cancers. An independent evaluation of the solution, which was used by GPs
in Leicestershire, demonstrated a 30% reduction in inappropriate referrals and a significant improvement
in the quality of referral documentation, as determined by cancer specialists.
• NHS Direct Health and Symptom Checker – delivered patient-facing online health and symptom
checking CIGs via a website, syndication feeds, and mobile apps. In the first year the use of this service
resulted in 1.3M fewer visits to other healthcare services, including 0.7M visits to GPs, saving the NHS
287887/2021/DM&A
532 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
52
an annual total of £57M1. The solution also detected the second wave of the H1N1 pandemic in the UK
in late 2010. More than 65 million users utilized the solution over 5½ years, with no reported clinical
incidents.
• NHS24 telephone triage solution – is operating with Arezzo® live in Scotland since May 2016. The
solution delivers CDS to clinical and non-clinical call handlers. It has been used to support more than 2
million calls. The solution integrates more than 300 separate CIGs into a unified call journey, utilizing
Arezzo‘s innovative ability to merge CIGs. Call times have been reduced and a recent evaluation has
shown that significantly fewer patients require GP appointments or visits after using the service.
• Great Ormond Street Hospital for Sick Children (London, UK) – where Arezzo® is being used to
manage complex diagnostic endocrine and metabolic pathways, integrated with the electronic medical
record. The solution has improved the safety and efficacy of the pathways execution, as well as saved one
FTE junior doctor equivalent in resources.
In New Zealand where the solution is integrated with electronic medical records for more than 90% of family
physicians. It has been used for more than 8 years. New Zealand has one of the highest rates of childhood
asthma in the world, with 20% of children affected. In 2009 BPAC NZ established a Childhood Asthma
programme for GPs which included Arezzo® CDS to help record patient histories and give advice on
medication/hospitalisation. There was an average of 25.5 hospitalisations per 100 child-years prior to their
first assessment with the BPAC module, and an average of 12.1 hospitalisations per 100 child-years following
assessment, i.e. 2.1:1 hospitalisation before compared with after (95% CI = 1.2 to 3.7, p=0.01). At the same
time in Australia, Arezzo® CDS was used by the largest health insurance provide to support a national
workplace health solution. The solution resulted in a significant improvement in the quality of
recommendations, as well as the efficiency of the service.
287887/2021/DM&A
533 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
53
Annexure 2: Survey Questionnaires
For ASHAs:
General Details
Role ASHA Facility Name:
Age Group < 20yrs 20-45 yrs > 45yrs Location
Gender Male Female No. Years of Work
Experience
Language
Proficiency
Can Read-write and
understand well
Can read-write little
but understand well
Cannot read-write.
Understand well.
Cannot read-
write. Understand
little.
Cannot read-write.
Cannot
understand.
Hindi
English
Other:
Which language above:
# Questions Response
1.
How much time do you spend in advising women on
reproductive health, pregnancy and taking care of
young children including immunizations?
<5
min
5-10 min
1
0
-
1
5
m
in
15-30 min >30 mins
2. How many households to you visit weekly for above? <10 10-25 25-50 50-75 >100
3. Do you maintain records for above?
Yes
No
4.
Is recording manual or electronic?
Manual Electronic
5.
When a new treatment guideline is released, are you
given any training
Yes No
6.
If yes, what is the quality of training: Very Poor
1
Poor
2
Neutral
3
Good
4
Excellen
t
5
7. How many hours are allotted for training?
8.
When new guideline training is given, rate your
experiences with respect to:
Very Poor
1
Poor
2
Neutral
3
Good
4
Excellen
t
5
Retention of knowledge
Ease of recollecting
Ease of reading/referring to the manual
Ease of reaching out to doctor
9.
For making any recommendation, do you
contact/refer to
Doctor/Nurse Manual Both
10.
How often do you refer to the guideline after its
release/training?
Never
Only
once
2-3 times
As and
when
required
Always
till I am
well
verse
with it
11. Do you use any digital tool for daily routine? Yes No
12.
If yes, what is the name and its purpose?
(probe around: name, what is it for, integration with
daily routine, user experience, how does it help,
limitations)
13.
If there is a tool which will help you in making clinical
decisions/recommendations by personalized data
analysis based on guidelines, will it be helpful?
Yes No May Be
287887/2021/DM&A
534 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
54
General Details
14.
What could incent you to use a tool like this? (money,
time saving, help in becoming more
knowledgeable….)
15.
Which phone or tablet, if any do you use and since
how long?
(Type of handset: Feature/smartphones)
16.
How will you rate yourself in terms of:
Very Bad
1
Bad
2
Average
3
Good
4
Very
good
5
Reading from a mobile device/tablet
Typing into a mobile device/tablet
Navigating through multiple screens
17.
Any other suggestions and Feedback you would like
to provide
18.
What data do you need to provide on a regular basis
to the center? If this app can help collect data
electronically for automatic upload, will it be useful?
287887/2021/DM&A
535 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
55
For Doctors and Nurses
General Details
Role
(circle one)
Medical Officer / GNM / ANM Facility Name:
Age Group < 20yrs 20-60 yrs > 60yrs Location
Gender Male Female No. Years of Work
Experience
Language
Proficiency
Can Read-write and
understand well
Can read-write little but
understand well
Cannot read-write. Understand
well.
Cannot read-write.
Understand little.
Cannot read-write.
Cannot understand.
Hindi
English
Other:
Which language above:
# Questions Response
1.
How many patients on average do you see
daily?
<10 10-25 25-50 50-75 >100
2.
What is the average time spent in one
consultation (symptoms, diagnosis, taking medical
history, checking past medical reports, prescribing
medicine)
<5 min 5-10 min 10-15 min 15-30min >30 mins
3. Do you record patient’s past medical history?
Yes
No
4. Is recording manual or electronic? Manual Electronic
5.
How is the recording done?
(Probe around: registers, process, storage, retrieval, etc). If
electronic, tablet, mobile, computer).
6.
How is the quality of records in terms of:
Very Poor
1
Poor
2
Neutral
3
Good
4
Very Good
5
Completeness of the record
(All data fields are captured)
Correctness of the records
(All the information is correctly captured)
Legibility of the records
(Can read and understand the records to act)
7.
Continuing with Q7, what is the reason for poor
record quality (If Q7 score is 2 or less than that)
8.
Do you know what standard treatment
guidelines (STGs) are?
For Medical Officers:
9.
When a new treatment guideline (STG) is
released, how much time does it takes to study
and remember the guidelines?
10.
How often do you refer to the guideline after its
release?
Never
Only
once
2-3 times
As and
when
required
Always till I
am well
versed with it
11.
How do you monitor the STG adherence of your
staff (nurse/ANM/ASHA)?
(probe around – Who does it, frequency, is this data
documented, what action is taken compliance is
low/poor)
For Nurses:
287887/2021/DM&A
536 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
56
General Details
12.
When a new treatment guideline is released,
are you given any training
Yes No
13.
If yes, what is the quality of training: Very Poor
1
Poor
2
Neutral
3
Good
4
Excellent
5
14.
How many hours are allotted for training?
15.
When new guideline training is given, rate your
experiences with respect to:
Very Poor
1
Poor
2
Neutral
3
Good
4
Excellent
5
Retention of knowledge
Ease of recollecting
Ease of reading/referring to the manual
Ease of reaching out to doctor
16.
For making any clinical
decisions/recommendations do you
contact/refer to
Doctor Manual Both
17.
How often do you refer to the guideline after its
release/training? Never
Only
once
2-3 times
As and
when
required
Always till I
am well
verse with it
For everyone:
18.
Do you use any digital tool for daily routine?
Yes No
19.
If yes, are you comfortable using this tool?
Yes No
20.
If yes, which tool and for what purpose?
(probe around: name, what is it for, integration with daily
routine, user experience, how does it help, limitations)
21.
If there is a tool which will help you in making
clinical decisions/recommendations by
personalized data analysis based on guidelines,
will it be helpful?
Yes No May Be
22.
What benefits do you foresee with the above
tool in your daily routine?
23.
What according to you will promote its uptake
among the other doctors/nurses/ASHA workers?
24.
Which phone/tablet do you use and since
when?
25.
How will you rate yourself in terms of? Very Bad
1
Bad
2
Average
3
Good
4
Very good
5
Browsing the internet
Typing/texting on phone or tablet
26.
Any other suggestions and Feedback you would like to provide:
27.
What data do you need to provide on a regular
basis to the center? If this app can help collect
data electronically for automatic upload, will it be
useful?
287887/2021/DM&A
537 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
57
Final Scoreard:
Please rate the following about your experience with the clinical decision support tool:
Strongly
Disagree
1
Disagree
2
Neutral
3
Agree
4
Strongly
Agree
5
Overall, I was satisfied with the CDS
tool
Usability
1 It is easy to use
2 The information is organized and
displayed in a logical manner
3 Recording patient data and
navigating through the workflow is
seamless
4 I am able to access the particular
guidelines as and when I need
them.
5 I feel comfortable using this tool in
social/community setting
6 The tool integrates/fits easily into
my daily routine
7 I am able to accomplish my tasks
more easily by using this tool
Usefulness
1 My productivity has increased
2 The quality of my interactions has
reduced due to the time spent in
entering data and reading
instructions from the tool
3 The toolgives me more information
on what toadvise patients than I
possess
4 I am more confident when I talk to
patients about their conditions and
recommendations
5 I am able to accomplish my tasks
more quickly
6 This tool helps me remember all
diagnostic procedures to be
advised
7 This tool helps me make better
clinical decisions
8 This tool helps me administer the
right drug at the right dose
9 This tool enables me to determine
which patients should be referred
10 I often need to override the
suggestions made by the tool
11 I would like to continue using this
tool for my daily routine and work
Other Comments:
287887/2021/DM&A
538 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
58
Annexure 3: Baseline Survey Results
ASHAs
The average age of the 20 female ASHAs was 29.4 years and their average experience was 5.4 years.
All 20 (100%) of them could read, write, and understand Hindi well, but only 3 (15%) could read, write, and
understand English well. 13 (65%) could not read or write English, but understood a little, whereas 2 (10%)
could not read, write nor understand English. It was thus evident that Hindi was the appropriate medium of
instruction and not English.
40% of the ASHAs visited less than 10 households a week,
30% visited between 10-25 households and 30% visited
between 25-50 households.
The average time spent on advising women on reproductive
health, pregnancy and childcare including immunizations was
10-15 minutes.
All 20 (100%) of the ASHAs maintain manual records in the
form of registers that they have access to for tracking
individual or family health. The registers have separate
sections for maternal and child health, where they record
current symptoms as narrated by the beneficiaries.
All 20 (100%) of the ASHAs knew what Standard Treatment Guidelines (STGs) are. 7 (35%) of the ASHAs
were trained on STGs one year ago, 3 (15%) were trained six months ago, 9 (45%) were trained less than
three months ago and 1 (5%) was trained less than one month ago.
0
5
10
15
20
Can Read-write
and understand
well
Can read-write
and understand
a bit
Cannot read-
write.
Understand well
Cannot read-
write.
Understand
little.
Cannot read-
write. Cannot
understand
Language Proficiency
ASHA
HindiEnglish
40%
30%
30%
Weekly Household Visits
ASHA
less than 1010-2525-50
287887/2021/DM&A
539 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
59
There is a wide variation on the hours spent on training - while 50% of the ASHAs responded saying they
spent 3-5 hours, 40% spent 1-2 hours and 10% spent 2-3 hours.
However, majority of the ASHAs (80%) perceived this training on STGs to be excellent.
As for the level of confidence in the field after training – 55% of the ASHAs felt a little confident and 45%
felt very confident, and none of them reported having low or no confidence.
7
3
1
7
12
months
6 months
3 months
less than
3 months
Last time ASHAs were
trained on STGs
1 1 3 1 2
5
6
1
0
2
4
6
8
9 8 7 6 5
1 or 2
hardly any sometimes
Number of training sessions received
on STGs
40%
10%
50%
Hours Spent on Training by ASHAs
3-5 hours2-3 hours 1-2 hours
80%
10%
10%
Qualty of STG Training as per
ASHAs
ExcellentGood Average
45%
55%
ASHAs' level of confidence in the field
after training
Very ConfidentA little confident
287887/2021/DM&A
540 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
60
Post new guideline training all the 20 (100%) of the ASHAs responded that they refer to the STGs as and
when required. The ease of referring to paper-based manuals was perceived to be good by 55%, excellent by
30%, and average by 5%, while 10% were not sure of this.
Their overall perceived retention of knowledge was good (95%), but the ease of recollecting was divided
between neutral (50%) to good and excellent (25% each). The ease of reaching out to doctors was perceived
as good (100%).
All 20 (100%) of the ASHAs responded positively to receiving handholding on the field. In case they are not
clear about STGs, 80% of them ask questions about STG and implement guidelines based on their
understanding, 15% of them are only mobilizing patients to health centres. All 20 (100%) of the ASHAs
consult with either the Medical Officer, ANM or the Nurse for giving any care advice, treatment, or referral.
None of the ASHAs used any digital tools for their daily routine at the time the base-line survey was
conducted. 60% of the ASHAs responded ―don‘t know‖ when asked if it would be helpful to have a digital
tool that helps them make recommendations through personalized data analysis based on STGs. 30%
responded positively and 10% responded ―may be‖. Thus it was evident that the level of awareness to the
potential of digital health was low, however there were 6 ASHAs who responded positively prior to the Phase
2 kick-off and the hypothesis was that this number would increase post the phase 2 study completion.
0
5
10
15
20
25
Retention of
knowledge
East of recollectingEase of
reading/referring to
the manual
Ease of reaching out
to the doctor
ASHAs experience with new Guideline Training
Very PoorPoorNeutralGoodExcellent
80%
15%
5%
Mechanisms to clear doubts about STGs
Ask questions about STG and
implement based on
understanding
Only bring patients to health
centers
Blank
287887/2021/DM&A
541 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
61
The biggest perceived benefits to use such a digital tool were to save time and increase their knowledge, with
9 (45%) of the ASHAs responding positively to this. However, 11 (55%) of the ASHAs had perceived
difficulties using such a tool, including 3 who thought it was not possible to use this tool at the site.
Only 5 (25%) of the ASHAs used smart phones and none of them had used a tablet device. Majority of the
ASHAs rated themselves average or below average in terms of various functionalities of mobile devices or
tablets.
All 20 (100%) ASHAs responded positively when asked If an app can help them collect data electronically for
automatic upload.
Subjectively, the suggestions and feedback of ASHAs included:
“Consulting ANM or doctor is what I always do. I am ready to get trained and work on the tablet.”
“Sometimes it is a bit cumbersome to transfer the patient to the doctor as we have to wait for them and sometimes the patient
denies to go, it takes a lot of time to convince them.”’
“I love the nature of my work and feel excited and happy doing it.”
“I have never used tablets and never entered data electronically. So I am excited to look forward for the training on the tablet and
then implement it in the field.”
30%
10%
60%
If there is a tool which will help you in making
any recommendations by personalized data
analysis based on guidelines, will it be helpful?
YesMaybeDon't Know
0
5
10
15
20
Very Bad Bad Below
Average
Average Good Very Good
Mobile device proficiency self-rating by ASHAs
Reading from a mobile device/tabletTyping into a mobile device/tablet
Navigating through multiple screens
287887/2021/DM&A
542 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
62
“Consulting ANM or doctor is always satisfactory for us and I am very positive toward the role I have been given and I find no
difficulty in doing the same. I am ready to get trained and work on the tablet. I think this will take less time as against the
manual method we use.”
“My suggestion would be to provide features of camera, watsapp in the tablet so that if there is no network atleast we can click the
photo of the information of the patient and later feed the information into the tablet. Alsowatsapp would help in sending the
information to the higher authorities.”
ANMs
The average age of the 21 female ANMs was 30.3 years and their average experience was 3.5 years.
All 21 (100%) of them could read, write, and understand Hindi well; 17 (81%) could read, write, and
understand English well, and 4 (19%) could understand English well, but could read and write a little. It was
thus evident that English proficiency was much higher in ANMs as compared to ASHAs.
2 ANMSs (9.5%) saw between 10-25 patients daily in the preceding week, 14 ANMs (67%) saw between 25-
50 patients per day, whereas 3 (14%) saw between 50-75 patients per day and 2 (9.5%) saw more than 100
patients per day in the preceding week.
35% of these patient interactions took 10-15, minutes
40% took 15-30 minutes and 25% took more than 30
minutes. The average time spent on advising women on
reproductive health, pregnancy and child-care including
immunizations was 10-15 minutes.
While 20 out 21 ANMs maintain manual medical
records of beneficiaries in registers, one ANM maintains
records both manually and electronically on the phone.
ANMs maintain manual records in the form of two
registers - ANC and master registers – that are stored in
the almirah of the office and can be accessed any time
for tracking individual or family health.
All 21 (100%) of the ANMs knew what Standard
Treatment Guidelines (STGs) are. 7 (33%) of the ANMs
were trained on STGs one year ago, 4 (19%) were
trained six months ago, 7 (33%) were trained less than
three months ago and 1 (4%) did not remember.
0
10
20
30
Hindi
English
Language Proficiency
ANM
Can Read-write and understand wellCan Read-write little and understand well
0.0%
10.5%
73.7%
15.8%
Daily patient load in preceding
week for ANMs
less than 1010-2525-5050-75
287887/2021/DM&A
543 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
63
ANMs receive training manuals which they use as and when needed for self-study after the training sessions.
4-5 hours are allotted to re-training or post-training support. Majority of the ASHAs perceived this training
on STGs to be good (62%) or excellent (32%).
71% of the ANMs hardly refer to these guidelines once published and training concluded, 19% as and when
required, and only 10% refer to the guidelines always until they are well versed with it.
ANMs overall perceived retention of knowledge, ease of recollecting, ease of referring to the manual were all
good (100% each).
However, on the ease of reaching out to Doctors 15 (71%) were neutral and 5 (28%) rated this as good,
showing definite opportunities for improvement in access to Doctors for ANMs, especially given the fact that
71% of them reach out to Doctors for making any clinical decisions/recommendations.
0
1
2
3
4
5
6
7
1 year ago
6 mos ago
3 mos ago
less than 3 mos ago
Not trained
7
4
2
7
0
Last time ANMs were trained on STGs
0
5
10
15
20
25
ANMs experience with training
on newly published guidelines
Retention of knowledge
East of recollecting
Ease of reading/referring to the manual
Ease of reaching out to the doctor
19%
10%
71%
Frequency of referring to the
guidelines after release/training?
As and when required
Allways till I am well versed with it
hardly anytime
287887/2021/DM&A
544 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
64
Only one ANM reported using a digital tool, but all 21 of
them are comfortable using digital tools. 62% of the
ANMs responded ―may be‖ when asked if it would be
helpful to have a digital tool that helps them make
recommendations through personalized data analysis based
on STGs. 5% responded positively and 10% responded
―may be‖.
16 (76%) of the ANMs used smart phones and 5 (24%)
used feature ‗dumb‘ phones. None of them used a tablet
device. 14 (67%) of the ANMs rated themselves good in
terms of various functionalities of mobile devices or
tablets, whereas 4 (19%) rated themselves are very bad.
Subjectively, the suggestions and feedback of ANMs included:
“Training sessions are very much helpful for us. When we find difficulty, we ask the trainers. Also, we look at the guidelines and
material given to us during training.
“There is a weekly AAA meeting i.e. ASHA, Anganwadi and ANM from 9 to 4 pm. It is scheduled for every Thursday and
all the weekly records (pregnant women and 0-5 years children) of ANM are tallied with the ASHA's records. All the ASHA
diaries are checked. The list of children which are due for vaccination are also discussed.”
“I would love to work on the tablet if training is given. I find training very useful for execution of our work.”
“In case of vaccination day, we are very busy. There is no private space to take the fundal height.”
Medical Officers
There were six male MOs and 3 female MOs who participated. All 9 of them could read, write, and
understand English and Hindi well.
Four MO‘s saw between 50-75
patients daily in the preceding
week, 4 MOs saw more than 100
patients per day, whereas 1 MO
saw between 25-50 patients per
day in the preceding week.6 MO‘s
took 10-15 minutes per patient
interaction whereas 3 MO‘s took
15-30 minutes.
All 9 MO‘s maintain manual medical records of beneficiariesincluding current symptoms as narrated and
additional screening questions on common and relevant illnesses. None of them used any digital tools for
daily routine work, however all of them were comfortable using digital tools.
(What benefits do you foresee with the above tool in your daily routine?)
All 9 MO‘s (100%) thought it would be helpful if there is a tool which will help them in making any
recommendations through personalized data analysis based on guidelines.
However, 8 of the MO‘s thought the use of the tablet-based digital health tool was questionable and 1
responded ‗don‘t know‘ to the question about promoting the uptake of such a tool amongst
doctors/nurses/ASHA workers.
1
4
4
Daily patient load in
preceding week for MOs
25-5050-75more than 100
6
3
Average time required per patient
interaction
10-15min15-30min
287887/2021/DM&A
545 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
65
All 9 MO‘s knew what Standard Treatment Guidelines (STGs) are and take 15-30 minutes to study and
remember it and then refer to it as and when required.
6 of the MOs rated themselves good in terms of various functionalities of mobile devices or tablets, whereas
2 of them rated themselves as very bad.
The MO‘s overall feedback was evident from their responses for suggestions - 4 were not interested, 3 were
neutral, 1 could not say if it would work and 1 was interested. The feedback from MO‘s also was that their
schedules were very busy especially with COVID-19 duties.
0
2
4
6
8
1 6 0 0 0 1 1
Mobile device proficiency self-rating by MO's
Typing into a mobile device/tablet
Navigating through multiple screens
Any other suggestions and Feedback you would like to provide
287887/2021/DM&A
546 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
66
Annexure 4: Phase 1 testing case mix
Maternal Cases (Retrospective from record review during Lockdown Period) n=82
No. of Journeys 357 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases = 82
ANC-without any complications 32
ANC with 1 or more
complications
2. Anemia in Pregnancy
3. Preeclampsia/Hypertens
ion Eclampsia
4. Gestational diabetes
34
5. 10 *
6. 16 **
7. 11
Post-natal 16
* These 10 women with anemia included 7 with only anemia and 3 women with anemia and preeclampsia
** These 16 women with preeclampsia included 13 women with only preeclampsia and 3 women with
preeclampsia and anemia.
Paediatric Cases (Retrospective from record review during Lockdown Period) n=73
No. of Journeys 354 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases
(n=73)
Feeding and Immunization alone
5 *
Pneumonia
Pneumonia alone
With feeding and immunization
7
5
2
Diarrhoea
Diarrhoea alone
Diarrhoea with Sick Young
Infant
With sick Child
With pneumonia
With feeding and immunization
32 **
18
5
3
2
4
Care of the Sick child
Care of the sick child alone
With feeding and immunization
18
9
9
Sick Young Infant 11
* The actual cases assessed for this guideline were 20 (5 normal, 9 sick child, 2 with pneumonia, 4 with
diarrhoea)
287887/2021/DM&A
547 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
67
Maternal Cases done (Prospective): from OPD (English version) (n=71)
No. of Journeys 342 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases (n=71)
ANC- without any complications 21
ANC with 1 or more
complications
Anemia in pregnancy
Hypertension/Eclampsia
Gestational diabetes mellitus
39
19 *
12**
12***
Post-natal 11
* These 19 include 16 with only anemia and 3 with anemia and GDM
** These 12 include 11 with only Preeclampsia and 1 with preeclampsia and GDM
*** These 12 include 8 with only GDM, 3 with GDM and anemia and 1 with GDM and preeclampsia.
Paediatric cases done (Prospective): from OPD (English version) n=102
No. of Journeys 504Continuum of Care Journeys completed for ASHA ANM Medical Officer
Cases
(n=102)
Feeding and Immunization 23*
Pneumonia
Pneumonia alone
With feeding and immunization
7
5
2
Diarrhoea
Diarrhoea alone
Diarrhoea w Sick Young Infant
With Sick child
With pneumonia
With feeding and immunization
32 **
12
1
3
3
13
Care of the sick child
Care of the sick child alone
With feeding and immunization
15
12
3
Sick Young Infant 14
New-born Care 11
*The actual cases assessed for this guideline were 41 (23 normal, 3 sick child 13 with diarrhoea and 2 with
pneumonia)
287887/2021/DM&A
548 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
68
Maternal cases done (Prospective): from OPD (English and Hindi version) n= 55
No. of Journeys = 264 (continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases
(n=55)
ANC-without complications
10
ANC with 1 or more complications
Anemia in Pregnancy
Preeclampsia/Hypertension/Eclampsia
Gestational diabetes mellitus
32
13*
10**
10
Post-natal
13
*These 13 include 12 cases of only anemia and 1 with anemia and preeclampsia
** These 10 include 9 cases of only preeclampsia and 1 with preeclampsia and anemia
PaediatricCases done (Prospective): from OPD (English and Hindi version) (n=56)
No. of Journeys 351(continuum of Care Journeys completed for ASHA ANM Medical Officer)
Cases (n=56)
Feeding and Immunization 5*
Pneumonia
Pneumonia alone
With feeding and immunization
10**
1
9
Diarrhoea
Diarrhoea alone
Diarrhoea with SYI
With Sick child
With Pneumonia
With feeding and immunization
12
1
4
0
0
7
Care of the sick child
Care of the sick child alone
With feeding and immunization
11
1
10
Sick Young Infant 7
New-born Care 11
* The actual cases assessed for feeding and immunization were 31(5 normal, 10 sick child, 9 with
pneumonia,7 with diarrhoea)
287887/2021/DM&A
549 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
69
The breakdown of the number of journeys is detailed below:
Maternal
Total number of Cases: 208
Total number of Journeys: 963
Journeys Breakdown: ASHA ANM MO
ANC-w/o complications 63 63 63
ANC w complications 105 105 105
ANC in Pregnancy 42 42 42
ANC-PE/HTN/EC 38 38 38
ANC-GDM 33 33 33
Post-natal 40 40 40
Pediatric
Total number of Cases: 231
Total number of Journeys: 2172
Journeys Breakdown: ASHA ANM MO
Feeding 92 92 92
Immunization 92 92 92
Pneumonia 29 29 29
Diarrhoea 76 76 76
Care of the sick child 50 50 50
Sick Young Infant 42 42 42
New born child 22 22 22
287887/2021/DM&A
550 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
70
Annexure 5: Interim Survey Results
A qualitative survey was executed in early March.
Usability
The overall positive response rate where respondents agreed or strongly agreed on the usability of CDSS-
integrated mobile app was 77.17%.
Usefulness
The overall positive response where respondence agreed or strongly agreed to useful features or disagreed or
strongly disagreed on reverse questions on the usability of CDSS-integrated mobile app was 72.95%.
2
3
1
2
0
2
6
4
2
4
5
6
4
8
3
6
4
1
19
20
26
23
22
23
15
11
14
11
15
14
05101520253035404550
6. I can accomplish my tasks more easily (मै …
5. It integrates/fits easily into my daily routine
(यह मेरी ददनचयाा मे आसानी में उपयुक्त …
4. I feel comfortable using it in
social/community setting (मै रोगी/ ऱाभाथी …
3. Recording patient data and navigating
through the workflow is seamless …
2. The necessary information and summary is
organized and displayed in a logical manner …
1. It is easy to use this mobile application (इस …
Usability N=46
Strongly DIsagree ??????Disagree ??????Neutral ??????Agree ??????Strongly Agree ??????
287887/2021/DM&A
551 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
71
82.61% of the respondents would like to continue using the app in their daily routine and the overall
satisfaction from the CDSS-integrated mobile app was 84.7%.
287887/2021/DM&A
552 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
72
Annexure 6: STG Adherence Secondary Data Analysis on limited journeys
Paediatric cases
The risk status of infants 0-2 months is based on birth weight, period of gestation, and their ability to
feed on day 1, and enables recommendation of an increased number of new-born home visits, and
additional care advice.
Assessment of the first feed was documented for 9 neonates. 100% reported successful breastfeeding
within 1 hour of birth.
First feed: (n = 9 neonates)
within 1 hour of birth 9
breastfed 9
suckled effectively 9
Gestational age at birth was documented for 75 infants, with 2 identified as high risk due to pre-term
birth:
Birth weight was recorded for 53% of infants aged 0-2 months:
One infant was identified as high risk, due to a birth weight < 2500g:
Across all age groups, a current weight measurement was documented at 11% of encounters:
287887/2021/DM&A
553 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
73
Weight-for-age status was automatically derived for children aged 2-60 months, according to WHO Child
Growth Standards. 16% of the children were identified as severely underweight, and 17% as moderately
underweight, for their age:
Symptom assessment for fever, pneumonia and diarrhoea-related diseases, and possible serious bacterial
infection in neonates, formed part of each encounter.
Temperature was measured at 15% of encounters:
If a temperature measurement was not documented, the FHWs were prompted to assess skin
temperature to touch. 19 cases of fever were documented: 4 based on a temperature measurement >=
37.5C; 15 based on assessment of the skin to touch.
5 cases of diarrhoea were documented across all encounters, with 1 recorded as persisting for 14 days or
more:
Diarrhoea (n = 1455)
acute (< 14 days) 4
persistent (>= 14 days) 1
with blood in stool 0
none 1450
None of the cases recorded significant signs of dehydration. A prescription of zinc was documented in 3
of the cases.
Symptoms of acute respiratory infection were assessed at encounters with children 2-60 months. 20 cases
of cough, breathing difficulty, or both were documented:
Symptom (n = 1320)
cough 19
breathing difficulty 3
none of the above 1300
287887/2021/DM&A
554 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
74
Further assessment identified symptoms consistent with fast-breathing pneumonia in 4 cases, and severe
pneumonia in 2 cases:
Encounters with infants 0-2 months included documentation of signs consistent with possible bacterial
infection. The frequency of positive findings is shown in the below table:
Sign Positive n
bulging fontanelle 12 162
cry weak or stopped 12 140
severe chest indrawing 1 155
frequent vomiting 1 140
large boil 1 148
crying incessant 1 140
redness of umbilicus 1 148
pus around umbilicus 0 148
jaundice 0 148
skin pustules 0 148
moving less or not at all 0 147
ear draining pus 0 147
abdomen distended 0 147
The outcome, or endpoint, of an encounter falls under three categories: urgent referral (or inpatient treatment
if appropriate) for severe symptoms, non-urgent referral for further investigation, and general home care
advice.
Urgent referral was recommended for 3% of cases, and non-urgent referral in 34% of cases:
287887/2021/DM&A
555 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
75
A breakdown of the reasons for urgent referral is shown below. ―Possible sepsis‖ refers to the presence of 1
or more severe signs, or 2 or more mild signs, of infection in an infant aged 0-2 months.
Of the cases where urgent referral was recommended, 51% documented that the child was unable to attend.
Reasons reported included unavailability of transport (3 cases), family reason (6 cases), and other individual
reasons (10 cases).
1
1
1
1
1
1
1
1
2
2
2
3
3
7
23
0 5 10 15 20 25
moderately low weight for age
persistent diarrhoea
poor weight gain
possible serious bacterial infection
signs of very severe febrile disease
unable to drink or feed
vomiting everything
weight less than 1800g
persistent fever
signs of pneumonia
signs of severe pneumonia
fever with respiratory symptoms
low body temperature
severe malnutrition
possible sepsis
Reasons for urgent referral
287887/2021/DM&A
556 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
76
Maternal cases (antenatal)
LMP was recorded in all but 10 antenatal cases:
For the cases where LMP was documented, 90% had a date recorded. 10% of women did not know their
LMP date:
All cases where the LMP was documented also had the EDD documented, including for women who did not
know the LMP date.
Yes, 18
No, 19
Availability of urgent referral (n=37)
287887/2021/DM&A
557 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
77
Women who did know the LMP date were asked for their best guess of the period of gestation in months. 11
women had an ultrasound to confirm the period of gestation, which was calculated as: 17 weeks (2 cases); 30
weeks (3 cases); 35 weeks (1 case); 38 weeks (1 case); and 40 weeks (1 case).
Symptoms were assessed by ASHAs and ANMs. The number of times that questions were asked varied
depending on the period of gestation, etc. This reflected the ability of the AI solution to understand the
context in which the questions were being asked, including the role of the User:
The range of answers was extended for some questions. Rather than just ‗yes‘/‘no‘ responses, the concepts of
‗bothersome‘ and ‗not bothersome‘ were used to distinguish the impact of some symptoms:
Both women who reported bothersome pain had continuous symptoms: 1 had mild pain; the other woman
had severe pain. The three women who had bothersome vomiting did not have persistent symptoms.
The frequency of positive findings for the other symptoms is shown in the following table:
287887/2021/DM&A
558 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
78
Symptom Present
Jaundice (symptom) 0
Oedema 1 (generalised)
Rash/itching 0
Epigastric pain 0
Excessive thirst* 12
Increased urinary frequency 7
Dysuria 2
Polyuria 0
Reduced urine output 0
Vaginal bleeding 2
White vaginal discharge 0
Blurring of vision 0
Watery fluid PV 2
*None of the women who reported excessive thirst had glycosuria
Symptoms of anemia were assessed with questions about dizziness, palpitations, and fatigue:
Bothersome symptoms No. Hb levels
Palpitations 1 Hb: 10.1
Fatigue 3 Hb: 7.8; Other 2 cases were >10
Dizziness 4 Hb: 9; 11; 11; 12
Breathless at rest* 2 Hb: 9; 11
*No longer asked as symptom of anemia but included for comparison
The question on palpitations includes the option to rate the symptom as ‗not bothersome‘. The following
chart shows the hemoglobin levels distribution for 16 women:
The question on easily fatigued includes the option to rate the symptom as ‗not bothersome‘. The following
chart shows the hemoglobin levels distribution for 42 women:
287887/2021/DM&A
559 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
79
Previous obstetric history was recorded for 123 women. The following table shows the number of previous
pregnancies:
No. previous
pregnancies (n=123 women)
0 4
1 42
2 32
3 18
4 12
5 10
6 3
7 1
8 0
9 1
The number of previous children is recorded in this table:
No. previous children (n=123 women)
0 8
1 40
2 32
3 19
4 13
5 8
6 3
Additional details of obstetric history are shown in the next table:
Obstetric history 'Yes'
Abortion 78
Previous Caesarean section 2
Previous complications 2
Previous neonatal death 1
Previous stillbirths 2
287887/2021/DM&A
560 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
80
Past medical history was recorded for 180 women:
Past medical history „Yes' No. asked
TB 1 180
Diabetes 2 180
Heart disease 1 180
Tobacco use and alcohol consumption was recorded for 180 women:
Social history „Yes'
Alcohol 1
Tobacco 16
Smoking tobacco 6
Smokeless tobacco 16
Family history was documented as completed for 180 women. There were no cases of hypertension, diabetes,
multiple pregnancies, TB, thalassemia, or mental retardation recorded in any families.
Obstetric risk assessment was performed in 213 cases. This is lower than the total number of antenatal cases
but reflects the fact that ASHAs do not perform risk assessments.
218 antenatal visits were performed by ANMs or MOs, which means that obstetric risk assessment was
recorded for 98% of the women seen by someone who is able to perform a risk assessment.
The proportion of high versus low obstetric risk cases is illustrated in the next chart, with the majority of
women (60%) having a high obstetric risk.
The details of the positive risk factors are set out in the following table, with several women having more
than one risk factor (209 risk factors distributed across 128 women):
High risk factors
No.
recorded
Anemia - mild 47
Anemia - moderate 40
Small-for-dates (good dating) 25
Smokeless tobacco usage 16
Previous neonatal loss 9
Foetal heart sounds absent 8
Large-for-dates (good dating) 8
Fundal height not increased 7
Smokers 6
287887/2021/DM&A
561 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
81
Fever 4
Easy fatigability 3
Previous difficult labour 3
Tachycardia 3
Abdominal pain 12 EDD 2
Breathless at rest 2
Headache 2
Pallor 2
Past history - diabetes 2
Past history - hypertension 2
Premature birth 2
Previous eclampsia 2
Small-for-dates (uncertain dating) 2
Alcohol consumption 1
Foetal heart sounds reduced 1
Generalised oedema 1
Other illness 1
Palpitations 1
Past history - TB 1
Past history - Heart disease 1
Previous delivery with congenital
anomaly 1
Previous pre-eclampsia 1
Rh negative 1
Vaginal bleeding <20 weeks 1
Vaginal bleeding >=20 weeks 1
219 women had body weight recorded in a visit, though 2 values were excluded as possible data-entry errors
(11 kg and 121 kg respectively). The following chart displays the distribution of maternal body weight, with a
median body weight of 48 kg:
287887/2021/DM&A
562 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
82
Maternal blood pressure was recorded in 221 visits. The distribution of systolic blood pressure readings is
displayed in the next chart. There is a bimodal distribution, suggesting an effect related to the measuring
device and/or the blood pressure measurement process:
All women who had elevated systolic blood pressure readings had repeat values taken during the visit. Two
women had persistent mild elevated blood pressure and were recommended for referral (see detailed Referral
Reasons section below).A similar bimodal distribution was seen for diastolic blood pressure readings.
Information was available on hemoglobin testing for 216 women. 197 women had hemoglobin
measurements. The distribution of maternal hemoglobin levels is shown below:
Blood group and Rhesus factor status was only measured in 3 women. In 174 visits, it was recorded that
blood group was not available.
VDRL testing was carried out in 174 women and recorded as ‗Not done‘ for 38 women. All tests that were
performed had negative results.
HIV testing was carried out in 196 women and recorded as ‗Not done‘ for 16. All tests that were performed
returned negative results.
Urine glucose was recorded as ‗ordered‘ in 179 women, with all results returning as negative for glycosuria. 34
women were documented as having the test ‗not ordered‘:
287887/2021/DM&A
563 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
83
Urine protein was recorded as ‗ordered‘ in 183 women, with all results returning as negative for proteinuria.
30 women were documented as having the test ‗not ordered‘.
Urine testing for glucose and albumin is carried out by ANMs and MOs. The tests are carried out in the
majority of visits:
Abdominal examination was carried out in 226 visits, with findings other than fundal height being recorded in
22 women. Nine women had an abdominal scar; 8 vertical and 2 ‗other‘. Foetal lie and presentation, as well as
foetal movements were also examined, depending on the period of gestation:
Foetal lie and presentation
Cephalic 67
Breech 2
Cannot tell 4
Foetal movements
Normal 110
Reduced 2
IFA therapy was reviewed in 225 women, with 8 women recorded as not taking IFA:
287887/2021/DM&A
564 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
84
Only one woman was recorded as having received parenteral iron therapy.
Tetanus toxoid vaccination status was reviewed in 187 women:
The AI solution calculated the scheduled date for a first tetanus toxoid dose in 43 women, for a second dose
in 3 women, and a booster date for 1 woman.
Deworming in the past 6 months was recorded as ‗done‘ for 123 of 201 women:not having treatment in the
second and third trimesters:
Deworming treatment was analyzed separately, with about 2/3rds of women having treatment in the first
trimester and 37% of women
287887/2021/DM&A
565 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
85
Counselling was recorded as being provided to 313 beneficiaries across the following topics. Note that
nutrition advice was provided on more than one occasion to some women:
Maternal cases (postnatal)
There were 44 women who were reviewed after delivery in February 2021. The demographic details have
been presented in a previous section of this document.
The following table shows the range of symptoms and other issues that were reviewed during the visits:
Postnatal review topics „Yes‟ „No‟
Regular diet started 18 11
Breast feeding baby 40 1
Lower abdominal pain 3 54
Breast feeding - insufficient milk 3 53
Breasts sore 1 54
Perineal pain 1 56
Breast lump 0 42
Breast tenderness 0 42
Breasts engorged 0 56
Breathlessness 0 57
Chills 0 56
Constipation 0 56
Contraception 0 24
Convulsions 0 42
Excessive bleeding at time of delivery 0 42
Dizziness 0 57
Epigastric pain 0 43
Easy fatiguability 0 41
Fever 0 56
Severe headache 0 43
Leg pain 0 55
Lochia abnormal 0 55
287887/2021/DM&A
566 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
86
Lochia foul-smelling 0 57
Mood changes 0 51
Nipples crusted or sore 0 57
Nipples inverted 0 55
Oedema 0 41
Pallor 0 41
Perineal swelling 0 57
Perineal tear 0 56
Urination burning 0 57
Urinary frequency increased 0 57
Urinary dribbling or retention 0 57
Uterus soft and tender 0 41
Vaginal bleeding 0 56
Blurring of vision 0 43
Body temperature of the mother was recorded during 52 visits. The following chart shows the distribution of
temperature in degrees Fahrenheit:
Blood pressure was recorded on 40 visits. The distribution of systolic and diastolic blood pressure values are
shown below:
Seven women had hemoglobin measured:
Hemoglobin measured
Yes 7*
No 10
Not known 27
*All 7 had mild anemia.
287887/2021/DM&A
567 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
87
Anemia treatment was recorded for 11 women; 31 women were noted as not receiving anemia treatment.
Calcium and vitamin D3 tablet usage was recorded in 57 visits:
Calcium Vit D3 tablets
One tab daily 35
Two tabs daily 21
No 1
The following advice was provided during 52 visits (91% of all visits):
Breastfeeding advice
Advice on calcium supplements
Family planning
IFA advice
Post-partum hygiene
Post-partum nutrition
Post-partum rest
Post-partum sexual activity
Family planning counselled was provided to 25 women. The number of women who agreed upon a family
planning method is shown in the following table:
Family planning method agreed upon
Yes 21
No 2*
Will decide later 2*
*No follow-up information is available for these women at the time of analysis
The family planning methods that the women agreed upon are listed in the next table:
Family planning methods
Centchroman* 9
Satisfied 6
Not satisfied 3
Condoms 12
Satisfied 12
COCP 0
ECP 0
IUCD 0
MPA 0
POP 0
*Centchroman is a family planning option developed in India
Referrals to a medical officer were recommended for the following reasons:
Recommended referrals to MO
Anemia 1
Suspected infection 5
Post-partum pain 3
___________________________________________________________________
287887/2021/DM&A
568 NITI Aayog, UK-DITHAIC Pilot Study : Project Report July 2021
88
287887/2021/DM&A
569