Choose Report Type
Publication Date
Report Upload
Download
(3.6 MB)
vertical
Energy
PDF Text
I
Development of enhanced energy data portal for India
Final Report -May 2O2L
Prayas (Enerry Group)
Final report of the research study on "Development of enhanced energy data
for lndia" sanctioned through letter no. 0-15072/29/I7-Research dated 30th
November 2OL7 from NlTl Aayog.
The study was carried out with the financial support of NlTl Aayog,
Government of lndia and conducted by Prayas (Energy Group), Pune
Pr.]as (tu€,gy 6roup)
T4K Development of enhanced energy data portal for India
Final Report-May 2O2l
Prayas (Energy Group)
Final report of the research study on "Development of enhanced energy data
for lndia" sanctioned through letter no. O-L5012/29/I7-Research dated 30th
November 2017 from NlTl Aayog.
The study was carried out with the financial support of NlTl Aayog,
Government of lndia and conducted by Prayas (Energy Group),
pune
14( Proiect duration
December 2017 to December 2019
Timeline of deliverables
Disclaimer
Prayas (Energy Group), Pune, has received the financial assistance under the Research Scheme of
NlTl Aayog to prepare the report. While due care has b,een exercised to prepare the report usin8 the
data from various sources, NlTl Aayog does not confirm the authenticity of data and accuracy of the
methodology to p repa re the re port. N lTl Aayog sha ll not be held respons ible fo r finding or opinions
expressed in the document. This responsibility completely rests with Prayas (Energy Group), Pune.
December 2017 First tranche of grant amount received by Prayas (Energy Group)
August 2018 Submitted first progress report indicatinB initial design decisions
October 2019 Progress update to Additional Secretary (Energy), NlTl Aayog
December 2019 Enhanced Dashboards development completed and ready for hand over
lanuary 2020
Handover and technicaltraining conducted by Prayas (Energy Group) for
Energy Division, NlTl Aayog and NIC
May 2020 - April 2021
Support to NlTl Aayog and NIC for undertaking security audit and
importing recent year data
Aptil L2 2021
Public launch of enhanced lndia Energy Dashboards by Vice Chairperson,
NlTl Aayog 1. Introduction and purpose
Availability of official data in the public domain is crucial for formulation, analysis and research of
public policy. ln addition to policy making, data availability is crucialfor decision making in
businesses and investment firms.
This is particularly true of the energy sector, where technical complexity is high and hence less
accessible to the general public. ln addition, given the globally interconnected nature of the sector
and wide uncertainties in future trends of availability and price of energy commodities, availability of
energy data in the public domain helps democratise the sector. Such data includes information such
as available reserves, trends in production and prices, patterns of consumption, and statistics related
to energy access.
The energy sector in lndia is administered by multiple ministries and departments at central and
state governments. At the central government, there are different ministries responsible for coal, oil
and gas, power, renewable energy and nuclear energy.
yet
there are inter-linkages between the
different sub-sectors and this has a bearing on how data is collected and organised. At the same
time, some ofthe administration is concurrent in nature between Centre and the states, most
prominently in the case ofelectricity. Given these considerations, collection and dissemination of
energy data is a rather complex activity and requires coordination across multiple levels. On the
consumption side, data is available from supply side ministries, household and enterprise
consumption surveys and from various ministries dealinB with various consuming sectors dealing
with transport and industry.
The 'lndia Energy Dashboards' portal hdsted by NlTl Aayog (henceforth referred to as the
Dashboards) was developed to make official energy data, available from multiple data agencies,
accessible in one place through an intuitive interface. The first version ofthe Dashboards was
launched in May 2017 and has been received well by lndian government agencies, analysts,
researchers, modellers, businesses and the general publicl.
Keeping in view this positive feedback, Prayas (Energy Group) undertook the development of
enhancements to the Dashboards with the support of a research grant from NlTl Aayog (vide
sanction order no. O-L5012/29117-Research dated 30th November 2017). These enhancements
include adding more data to the Dashboards, developing a workflow system to help maintain the
Dashboards, improving the download features on the Dashboards and providing a forum for data
users to interact.
The period ofthis research grant runs from December 14, 2017 to December 13,2019. This is the
final report describing the enhancements developed under the research grant.
2. Enhancements proposed
The following enhancements were proposed under the aforementioned research grant
1. Data at sub-yearly frequencies: currently the Dashboards only provide data on an annual
basis
1
The Dashboards has consistently received about 30 visits a day, of which 80-85% are from within lndia 2. A discussion forum for data users to facilitate discussions among data users regarding data
available on the Dashboards
3. A semi'automated workflow/ issue-tracking system for managing periodic updates to the
Dash boa rds by N lTl Aayog
4. Addition oftechnical and financial data of electricity utilities available from the regulatory
documents in one state
5. Enhanced data download: a cleaner, more intuitive way will be provided for downloading
data from the Dashboards
These features are described in greater detail below
3. Sub-yearly data
Two types of sub yearly data - published monthly data and live data from some apps and portals
maintained by government agencies - have been added to the Dashboards.
3.1. Monthly data
This data is sourced from the monthlv reports that are regularly published for the electricity,
petroleum and naturalgas sectors. Since monthly reports forthe coal sector are not published, they
have not been included in the Dashboards. Table l lists the monthly data'?added to the Dashboards.
Table L: List of monthly data added to the Dashboards
3.2. Data from live government portals
ln addition to published monthly reports, live data is published at daily, monthly and yearly
frequencies through various dashboards and mobile apps, primarily by centralgovernment
organisations. Wlth the proliferation of such dashboards in recent years, they have become a rich
source of data in the ener8y sector. lt would be useful to provide access to data from these portals in
one place. However, this requires publicly accessible APls for programmatically sourcing the data on
Source
oilP roduction State-wise crude oil productionMoPNG
oil Production Domestic production of petroleum products PPAC
oil Consu mptionDomestic consumption of petroleum products PPAC
oil lmport/exportlmport/export of crude oil and petroleum products PPAC
Naturalgas Production State-wise natural gas productionPPAC
Natural gas lm port lmport of LNG (liquefied naturalgas)
ElectricityCapacity State-wise, ownership-wise and source-wise installed
capacity
CEA
Electricity P roduction Energy generation, plant load factor and imports
Electricity Su pp ly State-wise peak and energy power supply position
Report
CEA
'?Note
that monthly data is provisional in nature. Hence monthly data reported on the Dashboards will be
provisional. This will be indicated on the Dashboards.
Sector lCategory |
Description
I
PPAC
CEA a regular basis. NlTl Aayog had written to the respective authorities for all available energy sector
portals, and APls have been provided for four of them - Saubhagya, UJALA,
pRAApTt
and Vidyut
PRAVAH - and the associated data has been added to the Dashboards. Table 2 lists the portals for
which APls are available along with a description ofthe data that is added to the Dashboards.
Table 2: List of portals whose data is added to the Dashboards
4. Regulatory data
Detailed performance and financial data is available on an annual basis from regulatory documents
such as tariff petitions filed by distribution and generation companies, and the orders issued by
regulatory commissions. As part ofthis proiect, regulatory data from the state of Maharashtra,
3
With the completion of household electrification, data under saubhagya has not been updated since March
2079.
Sector Category Portal (website)Description
E lectricityAccess 5au bhagya
(https://sa u bhagya.gov. inl)
Monthly state-wise progress of
intensive household electricity
connections drive under the
Saubhagya program that was
active from October 2017 to
March 20193.
E lectricityEfficiency UJALA (http://ujala. gov. inl) Dashboard containing state-
wise energy efficient appliance
sales data under the 'Unnat
Jyoti by Affordable LEDs for All'
(UJALA) program.
ElectricityGeneration PRAAPTI (https://praapti.inl)Dashboard with monthly status
of payments pending from
DISCOMS to Generators. Full
form: Payment Ratjfication And
Analysis in Power procu rement
for bringing Transparency in
lnvoicing of generators
(PRAAPTI).
ElectricityDistributionVidyut PRAVAH
(http://www.vidyutpravah.inl)
State-wise daily peak demand
and nation-wide shortage at the
time of peak demand. Although
the Vidyut PRAVAH dashboard
contains zone-wise price every
15 minutes, this data has not
been made available through
the APl.
I
I
I
I specifically for the area served by the state-owned distribution utility -
MSEDCL - has been added to
the Dashboards. Following is the list of data items added.
CategoryDescriptionData
Power Purchase Long term and medium
term power purchase by
ownership (state, central,
private) and
fuel/technology (coal,
solar etc). Some details
such as source-wise
disaggregation of
renewable energy
generation are not
available.
Capacity (MW)
Generation (MU)
Fixed charge (Rs/kWh)
Variable Charge (Rs/kWh)
Power Purchase Short-term purchases Quantum (MU)
Average Cost (Rs/kWh)
RPOSolar and non-solar RPO Obligation (%)
Met (%)
Sa lesConsumer category-wise
Consumption Open access consum ption
by duration and source
No. of consumers,
Total connected load (MW) and
Total consumption (MU)
SalesSale of surplus energy Quantum (MU) and
Average realised rate (Rs/kWh)
LossesTransmission and
distribution losses
lnter-state transmission losses (%)
lntra-state transmission losses (%)
Distribution losses (%)
Expenditure Expenditure broken down
under various heads
Total power purchase costs (Rs Cr)
Distribution capital expenditure (Rs Cr)
Distribution operating expenditure {Rs Cr)
Distribution other expenses (Rs Cr)
SubsidySubsidy booked and
received from government
Category-wise subsidy received (Rs Cr)
Total subsidy booked {Rs Cd
RevenueRevenue broken down
under various heads
Revenue from tariff (Rs Cr),
Non-tariff income (Rs Cr),
Sales migration (Rs Cr),
Sale of surplus (Rs Cr),
Subsidy receipts (Rs Cr),
Revenue Gap/Surplus for the year (Rs Cr),
Rate of carrying cost
1%)
I
No. of consumers,
Sales (MU),
Revenue (Rs Cr) and
Average billing rate (Rs/kwh)
I
I
I
I 6. Workflow system
Ongoing maintenance of the Dashboards is an important task and is necessary for keeping the
Dashboards up-to-date and relevant. A semi-automated workflow system has been developed to
help Nlrl Aayog put in place robust processes to assist in maintaining the Dashboards going forward.
Routine tasks for updating or adding data can be tracked through this system. The workflow system
performs basic validation checks thus helping avoid incorrect data entry on the Dashboards. The
system also provides for multiple users with different privileges and responsibilities, thus providing
Nlrl Aayog management effective oversight over maintenance of the Dashboards. This system will
be deployed on Nlrl Aayog's internal network. Nlrl Aayog staff would be provided hands-on training
,or how to use the workflow system to add new data and approve it so that it appears on the
Dashboards.
The following flowchart shows the process of using the workflow system to periodically update the
Dashboards.
Figure 1: Proc€ss of adding data to the lndia Energy Dashboards using the workflow system
NlTl sends blank data formats to data agencies (one time task)
Yes
Data
..--.Y:"'l
No
lndicators 5ummary indicators
derived from the above
parameters
Consumer category-wise:
Average power purchase cost (Rs/kWh),
Average cost of supply (Rs/kWh),
Average billing rate (Rs/kwh) and
Cross-subsidy (Rs/kWh)
Data files from
Agencies
lmport Data using
Energy Dashboard
Workflow System
Reject Data,
lnform Agency
Enerty Da5hboard Workflow
System loads data into database
Refresh Dashboard file using Tableau Desktop
Publish on Tableau Public/Server
Appears on lndia Energy Dashboards
I
5. Renewables menu
Given the increasing importance of renewables based electricity in the lndian energy system, it was
felt during a review meeting with NlTl Aayog management that a separate menu item for
renewables should be added to the Dashboards. Based on this, a new menu item was created, and
dashboards and charts relevant to renewables have been added to this section.
Following are a few screenshots of the workflow system. 6.1. Search page or home page
The landing page of the workflow system is a search interface which provides the user status
regarding which data items have been imported into the database, and which are pending. The
search results provide templates as well as sample past data that can be used to import pending
data. A snapshot of the search page is shown in Figure 2.
Figure 2:Search page
lmt-rt ld$on LoO iltrstB
r6pod siarus €j arr C)
p"nains !) p.*i,io".l O FiEl 8ur llot AppDred
cra,LlaDsaPtiso
cslac.!!mprion
ael c6nahP1im
&qrplr!! llEporl I 5!!]pls
]!!opli]!llo!!1 I s!!!rt
IcDplr& tlopei I 5,i!de
asolole llqrql I D.!.
6.2. Data import
The data import page allows the user to upload a file in .csv format consisting of data for the
corresponding year or month in the supplied template (see Figure 3). The imported data is checked
for correctness of categorical data such as states, generation sources and consuming sectors. lf these
checks fail, the user is provided a report of the reasons for failure, based on which the user can re-
import after correcting the data or adding new categories in the'Masters' page. ln addition, there
are configurable checks for the actual parametric data that is being imported, such as installed
capacity, quantity produced and quantity consumed. lf these checks fail, a warning is raised which
can either be overridden by the user. lf the data is determined to be wrong, it can be corrected and
re-imported.
The data that is imported can either be marked "Provisional" or "Final". "Provisional" data can be
overwritten multiple times, but "Final" data cannot be overwritten. The import of both "Provisional"
and "Final" data needs to be approved by a user with suitable privileges.
Hose Search Frgure 3: Datd import pdge
l{6t s lLr. tr.r! Domldd fbs APP
' D.t S.Lm.
caregory : c.nqrnption
:
?oli -
(,r,""r li'p,",i,r."a
o.o,se. , N! 6L el.<t.c.
6.3. Import log
The history of data imported to the database can be seen on the 'lmport Log' page. This page
provides a search facility to narrow down the history to specific sector and sub-sector. The search
results indicate the period for which data was imported, when the data was imported, whether it is
final or provisional, and which user imported the data (refer to Figure 4).
Figure 4: lmport Log
i'&,r '.l:::" r,ta.r'B n .t D.1, D.mr6d Frow App o'ra u*i
Drr. S!1.8. AdBin
El€d,,civ tklkiiyr@drh ;:
- !3n:?rek , L.nr,rorth
Ele.(i.ity aapelry f!..$ry;
Us Y.d Month 6n l hpon Dar.
2019
6,4. Fetching data from live government portals
As mentioned in Section 3.2, data from live government portals has been added to the Dashboards
This data is accessed through the APls that have been provided. The trigger for fetching the data is
provided through the workflow system under the tab 'APP Data' (see Figure 5). When the portal is
chosen and'Fetch Data' button is clicked, the data is fetched from the respective portal and added
to the database. This data is then available for download through the data download interface
described in Section 7. Figure 5: Fetching live portal data
Stafus
Output Type
ApiURLl
ApiURL2
t 5/01/2020 1al8:31
Data Processed Suc(essfully on 2020{1-'15 16118130
http/101.203 13a 1 tg,/Ujala .Vap,/.ommorlc€tNationaDarar'{D)
httpi//103.203.1 18 139,4riala_.Vapilommon/&tstateData{StatelD}?applian€etype={lD}
lD=2 rneans 8uib, lD=l means Fan, lD = 4 means Tube lighl
6.5. Generating the Energy flow [Sankey) and Energy balance tables
One of the im portant featu res of the Dashboards is to provide energy flow (Sankey) diagrams and
energy balance tables for the years for which final data is available. The workflow system provides
the ability to inspect if all yearly data is available for a particular year, and if the data exists, allow for
the energy flow diagram and energy balance table to be generated using the data. Figure 6 has a
snapshot of the same.
Figure 5: Generation of energy flow Sankey diagram and energy balance tables
'lmport lmpon Los Mr!L.3 lrcla D.l! O(MlsdAPP o'I. U3..
Adn'n
.
tlementr lor whi(h dnla is impo.tei {or This year
Y.a .rE D.lli*
Corl P.adudion Dra.ni(
aal rmp.r1 Co.ruoi.g scd.'
aelJlrrdu.li.rn Doin.stk St l€ .nd aompary wis.
(ft| rnrlr.,r!E9. !r.tor*r.
.Gl.on\ump!io. tr.re a.d ueag. sd.r wi{
aoal hr,.n an{.t.y 16.
Auxiliary cctrrlnptD. d.b
6,6, User administration -
'User admin' page
Accounts in the workflow system are assigned one of four roles -
user, approver, user admin and
developer admin - with a hierarchical set of privileges. lJser accou nts can upload data while the
approver accounts can approve the uploaded data and make changes to the master tables in
addition to uploading data- The user admin role can add, change or remove user accounts. The
developer admin account can undertake certain maintenance and development tasks such as
updating the meta-data and queries for data download. An 'Energy Data Portal and Application
Overview' document shall be provided at the time of deployment. This document consists details of
privileges accorded to each of the above roles.
t
Gln€iatoElrlancoDblii ..,tttu
2016 7. Improved data download
The first version of the Dashboards provides a way to download data in csv format through the
"Download" button at the bottom right corner provided by the Tableau@ data visualisation software
with which the charts are created. Data thus downloaded is not always clean, in that it may consist
of intermediate data used to produce the visualisations within Tableau. ln addition, this download
facility does not provide the user the ability to choose the category of data in line with the structure
of the Dashboards. As part of the current enhancements, a cleaner, more intuitive way of
downloading data from the portal is provided.
The improved data download interface can be accessed by NlTl Aayog staff from the workflow
system and by the general public through the lndia Energy Dashboards portal
{the blue button seen
in Figure 7). The user can choose the sector and category of data as organised on the lndia Energy
Dashboards. When the 'Download' button is pressed, the data is downloaded in a csv format. The
user can choose the level of detail by selecting either aggregate or itemised detail (see Figure 8).
The public version of the data download application will be enabled on the Dashboards when the
necessary lT security clearances are received from N lC.
Figure 7: Download data button on the Dashboards (top right corner, next to the help button)
NITIAayog INDIA' ''
,DASHBOARDS 8IILIS.r[
I'IDIA ETiERGY
Electricityconsumpton OOO
\+
tigure 8: Data download application interface
TLECTfu'ITYEl.lrrL. tr f,l.r:ll,l,/ CeBBna
g A!.ilahiil! !t Po*{
(!'
&e-e"t
(l
ncoi,"a
B. Feedback forum
The energy data user community could be engaged to help improve the quality of the data and
interface of the Dashboards. A feedback forum has been added to the Dashboards in order to
facilitate feedback from data users (see Figure 9). Figure 9: Feedback forum interface on the
Feedback & Sugqestlons topic: Overvi€w
0 Colnmenls lnda, Energy Dashbo.rd (ednrl 3 Oisqrs' Pnva€y Policy
ar Eecffifteni' tr6 IG@
'E nergy Balance'page
5 t-ogi, -
Stan the dis.russi.n. .
""
s'cn * yrtx o.so.,s @
i*(}#@
The feedback forum consists of separate sub-forums for each domain listed in the left hand side
menu on the Dashboards - Overview, Electricitv, Coal, Oil, Gas, State and References. This level of
granularity reduces clutter so that the feedback is relevant for the domain being viewed while also
ensuring that feedback is easier to track.
The feedback forum will be moderated by a respected energy sector researcher/analyst identified by
NlTl Aayog. Dr Rangan Baner.jee, Professor, llT Bombay, has agreed to moderate the forum.
9. Deployment and support
The enhanced Dashboards and the workflow application was deployed on NlTl Aayog servers and is
publicly accessible through the web at http://niti.sov.in/edm. The Dashboards were launched by the
Vice Chairperson, NlTl Aayog on Aprtl12'6 2O2!. The undedying database and Tableauo dashboards
were handed over to NlTl Aayog and NIC staif for future maintenance and feature addition. Training
was provided to NlTl Aayog and NIC staff at the time of hand over, and the various features were
explained along with a walk through of a typical workflow process.
Prayas (Energy Group) assisted NlTl Aayog staff over phone/audio conferences on an as needed
basis from May 2020 to April 2021. During this period, recent years' data has been incorporated into
the Dashboards by NlTl Aayog and data is available up to 2019-20, and in some cases up to 2O2O-21.
10. Future work
NlTl Aayog should institutionalise the data update activity so that the Dashboards continues to
remain relevant by having the latest available data.
ln addition, there is scope for significantly enhancing the amount of data and visualisations available
through the Dashboards. Following are some suggested areas of future work.
Regulatory data can be expanded to more states and DISCOMs using the provided formats
More live portals can be added by requesting for public Apls
The energy data user community can be engaged and based on the feedback received
o any data inconsistencies can be fixed and
o new data and dashboards can be added to lndia Energy Dashboards.
NlTl Aayog can consider a quarterly newsletter in which updates made to lndia Energy
Dash boards are commu nicated.
Development of enhanced energy data portal for India
Final Report -May 2O2L
Prayas (Enerry Group)
Final report of the research study on "Development of enhanced energy data
for lndia" sanctioned through letter no. 0-15072/29/I7-Research dated 30th
November 2OL7 from NlTl Aayog.
The study was carried out with the financial support of NlTl Aayog,
Government of lndia and conducted by Prayas (Energy Group), Pune
Pr.]as (tu€,gy 6roup)
T4K Development of enhanced energy data portal for India
Final Report-May 2O2l
Prayas (Energy Group)
Final report of the research study on "Development of enhanced energy data
for lndia" sanctioned through letter no. O-L5012/29/I7-Research dated 30th
November 2017 from NlTl Aayog.
The study was carried out with the financial support of NlTl Aayog,
Government of lndia and conducted by Prayas (Energy Group),
pune
14( Proiect duration
December 2017 to December 2019
Timeline of deliverables
Disclaimer
Prayas (Energy Group), Pune, has received the financial assistance under the Research Scheme of
NlTl Aayog to prepare the report. While due care has b,een exercised to prepare the report usin8 the
data from various sources, NlTl Aayog does not confirm the authenticity of data and accuracy of the
methodology to p repa re the re port. N lTl Aayog sha ll not be held respons ible fo r finding or opinions
expressed in the document. This responsibility completely rests with Prayas (Energy Group), Pune.
December 2017 First tranche of grant amount received by Prayas (Energy Group)
August 2018 Submitted first progress report indicatinB initial design decisions
October 2019 Progress update to Additional Secretary (Energy), NlTl Aayog
December 2019 Enhanced Dashboards development completed and ready for hand over
lanuary 2020
Handover and technicaltraining conducted by Prayas (Energy Group) for
Energy Division, NlTl Aayog and NIC
May 2020 - April 2021
Support to NlTl Aayog and NIC for undertaking security audit and
importing recent year data
Aptil L2 2021
Public launch of enhanced lndia Energy Dashboards by Vice Chairperson,
NlTl Aayog 1. Introduction and purpose
Availability of official data in the public domain is crucial for formulation, analysis and research of
public policy. ln addition to policy making, data availability is crucialfor decision making in
businesses and investment firms.
This is particularly true of the energy sector, where technical complexity is high and hence less
accessible to the general public. ln addition, given the globally interconnected nature of the sector
and wide uncertainties in future trends of availability and price of energy commodities, availability of
energy data in the public domain helps democratise the sector. Such data includes information such
as available reserves, trends in production and prices, patterns of consumption, and statistics related
to energy access.
The energy sector in lndia is administered by multiple ministries and departments at central and
state governments. At the central government, there are different ministries responsible for coal, oil
and gas, power, renewable energy and nuclear energy.
yet
there are inter-linkages between the
different sub-sectors and this has a bearing on how data is collected and organised. At the same
time, some ofthe administration is concurrent in nature between Centre and the states, most
prominently in the case ofelectricity. Given these considerations, collection and dissemination of
energy data is a rather complex activity and requires coordination across multiple levels. On the
consumption side, data is available from supply side ministries, household and enterprise
consumption surveys and from various ministries dealinB with various consuming sectors dealing
with transport and industry.
The 'lndia Energy Dashboards' portal hdsted by NlTl Aayog (henceforth referred to as the
Dashboards) was developed to make official energy data, available from multiple data agencies,
accessible in one place through an intuitive interface. The first version ofthe Dashboards was
launched in May 2017 and has been received well by lndian government agencies, analysts,
researchers, modellers, businesses and the general publicl.
Keeping in view this positive feedback, Prayas (Energy Group) undertook the development of
enhancements to the Dashboards with the support of a research grant from NlTl Aayog (vide
sanction order no. O-L5012/29117-Research dated 30th November 2017). These enhancements
include adding more data to the Dashboards, developing a workflow system to help maintain the
Dashboards, improving the download features on the Dashboards and providing a forum for data
users to interact.
The period ofthis research grant runs from December 14, 2017 to December 13,2019. This is the
final report describing the enhancements developed under the research grant.
2. Enhancements proposed
The following enhancements were proposed under the aforementioned research grant
1. Data at sub-yearly frequencies: currently the Dashboards only provide data on an annual
basis
1
The Dashboards has consistently received about 30 visits a day, of which 80-85% are from within lndia 2. A discussion forum for data users to facilitate discussions among data users regarding data
available on the Dashboards
3. A semi'automated workflow/ issue-tracking system for managing periodic updates to the
Dash boa rds by N lTl Aayog
4. Addition oftechnical and financial data of electricity utilities available from the regulatory
documents in one state
5. Enhanced data download: a cleaner, more intuitive way will be provided for downloading
data from the Dashboards
These features are described in greater detail below
3. Sub-yearly data
Two types of sub yearly data - published monthly data and live data from some apps and portals
maintained by government agencies - have been added to the Dashboards.
3.1. Monthly data
This data is sourced from the monthlv reports that are regularly published for the electricity,
petroleum and naturalgas sectors. Since monthly reports forthe coal sector are not published, they
have not been included in the Dashboards. Table l lists the monthly data'?added to the Dashboards.
Table L: List of monthly data added to the Dashboards
3.2. Data from live government portals
ln addition to published monthly reports, live data is published at daily, monthly and yearly
frequencies through various dashboards and mobile apps, primarily by centralgovernment
organisations. Wlth the proliferation of such dashboards in recent years, they have become a rich
source of data in the ener8y sector. lt would be useful to provide access to data from these portals in
one place. However, this requires publicly accessible APls for programmatically sourcing the data on
Source
oilP roduction State-wise crude oil productionMoPNG
oil Production Domestic production of petroleum products PPAC
oil Consu mptionDomestic consumption of petroleum products PPAC
oil lmport/exportlmport/export of crude oil and petroleum products PPAC
Naturalgas Production State-wise natural gas productionPPAC
Natural gas lm port lmport of LNG (liquefied naturalgas)
ElectricityCapacity State-wise, ownership-wise and source-wise installed
capacity
CEA
Electricity P roduction Energy generation, plant load factor and imports
Electricity Su pp ly State-wise peak and energy power supply position
Report
CEA
'?Note
that monthly data is provisional in nature. Hence monthly data reported on the Dashboards will be
provisional. This will be indicated on the Dashboards.
Sector lCategory |
Description
I
PPAC
CEA a regular basis. NlTl Aayog had written to the respective authorities for all available energy sector
portals, and APls have been provided for four of them - Saubhagya, UJALA,
pRAApTt
and Vidyut
PRAVAH - and the associated data has been added to the Dashboards. Table 2 lists the portals for
which APls are available along with a description ofthe data that is added to the Dashboards.
Table 2: List of portals whose data is added to the Dashboards
4. Regulatory data
Detailed performance and financial data is available on an annual basis from regulatory documents
such as tariff petitions filed by distribution and generation companies, and the orders issued by
regulatory commissions. As part ofthis proiect, regulatory data from the state of Maharashtra,
3
With the completion of household electrification, data under saubhagya has not been updated since March
2079.
Sector Category Portal (website)Description
E lectricityAccess 5au bhagya
(https://sa u bhagya.gov. inl)
Monthly state-wise progress of
intensive household electricity
connections drive under the
Saubhagya program that was
active from October 2017 to
March 20193.
E lectricityEfficiency UJALA (http://ujala. gov. inl) Dashboard containing state-
wise energy efficient appliance
sales data under the 'Unnat
Jyoti by Affordable LEDs for All'
(UJALA) program.
ElectricityGeneration PRAAPTI (https://praapti.inl)Dashboard with monthly status
of payments pending from
DISCOMS to Generators. Full
form: Payment Ratjfication And
Analysis in Power procu rement
for bringing Transparency in
lnvoicing of generators
(PRAAPTI).
ElectricityDistributionVidyut PRAVAH
(http://www.vidyutpravah.inl)
State-wise daily peak demand
and nation-wide shortage at the
time of peak demand. Although
the Vidyut PRAVAH dashboard
contains zone-wise price every
15 minutes, this data has not
been made available through
the APl.
I
I
I
I specifically for the area served by the state-owned distribution utility -
MSEDCL - has been added to
the Dashboards. Following is the list of data items added.
CategoryDescriptionData
Power Purchase Long term and medium
term power purchase by
ownership (state, central,
private) and
fuel/technology (coal,
solar etc). Some details
such as source-wise
disaggregation of
renewable energy
generation are not
available.
Capacity (MW)
Generation (MU)
Fixed charge (Rs/kWh)
Variable Charge (Rs/kWh)
Power Purchase Short-term purchases Quantum (MU)
Average Cost (Rs/kWh)
RPOSolar and non-solar RPO Obligation (%)
Met (%)
Sa lesConsumer category-wise
Consumption Open access consum ption
by duration and source
No. of consumers,
Total connected load (MW) and
Total consumption (MU)
SalesSale of surplus energy Quantum (MU) and
Average realised rate (Rs/kWh)
LossesTransmission and
distribution losses
lnter-state transmission losses (%)
lntra-state transmission losses (%)
Distribution losses (%)
Expenditure Expenditure broken down
under various heads
Total power purchase costs (Rs Cr)
Distribution capital expenditure (Rs Cr)
Distribution operating expenditure {Rs Cr)
Distribution other expenses (Rs Cr)
SubsidySubsidy booked and
received from government
Category-wise subsidy received (Rs Cr)
Total subsidy booked {Rs Cd
RevenueRevenue broken down
under various heads
Revenue from tariff (Rs Cr),
Non-tariff income (Rs Cr),
Sales migration (Rs Cr),
Sale of surplus (Rs Cr),
Subsidy receipts (Rs Cr),
Revenue Gap/Surplus for the year (Rs Cr),
Rate of carrying cost
1%)
I
No. of consumers,
Sales (MU),
Revenue (Rs Cr) and
Average billing rate (Rs/kwh)
I
I
I
I 6. Workflow system
Ongoing maintenance of the Dashboards is an important task and is necessary for keeping the
Dashboards up-to-date and relevant. A semi-automated workflow system has been developed to
help Nlrl Aayog put in place robust processes to assist in maintaining the Dashboards going forward.
Routine tasks for updating or adding data can be tracked through this system. The workflow system
performs basic validation checks thus helping avoid incorrect data entry on the Dashboards. The
system also provides for multiple users with different privileges and responsibilities, thus providing
Nlrl Aayog management effective oversight over maintenance of the Dashboards. This system will
be deployed on Nlrl Aayog's internal network. Nlrl Aayog staff would be provided hands-on training
,or how to use the workflow system to add new data and approve it so that it appears on the
Dashboards.
The following flowchart shows the process of using the workflow system to periodically update the
Dashboards.
Figure 1: Proc€ss of adding data to the lndia Energy Dashboards using the workflow system
NlTl sends blank data formats to data agencies (one time task)
Yes
Data
..--.Y:"'l
No
lndicators 5ummary indicators
derived from the above
parameters
Consumer category-wise:
Average power purchase cost (Rs/kWh),
Average cost of supply (Rs/kWh),
Average billing rate (Rs/kwh) and
Cross-subsidy (Rs/kWh)
Data files from
Agencies
lmport Data using
Energy Dashboard
Workflow System
Reject Data,
lnform Agency
Enerty Da5hboard Workflow
System loads data into database
Refresh Dashboard file using Tableau Desktop
Publish on Tableau Public/Server
Appears on lndia Energy Dashboards
I
5. Renewables menu
Given the increasing importance of renewables based electricity in the lndian energy system, it was
felt during a review meeting with NlTl Aayog management that a separate menu item for
renewables should be added to the Dashboards. Based on this, a new menu item was created, and
dashboards and charts relevant to renewables have been added to this section.
Following are a few screenshots of the workflow system. 6.1. Search page or home page
The landing page of the workflow system is a search interface which provides the user status
regarding which data items have been imported into the database, and which are pending. The
search results provide templates as well as sample past data that can be used to import pending
data. A snapshot of the search page is shown in Figure 2.
Figure 2:Search page
lmt-rt ld$on LoO iltrstB
r6pod siarus €j arr C)
p"nains !) p.*i,io".l O FiEl 8ur llot AppDred
cra,LlaDsaPtiso
cslac.!!mprion
ael c6nahP1im
&qrplr!! llEporl I 5!!]pls
]!!opli]!llo!!1 I s!!!rt
IcDplr& tlopei I 5,i!de
asolole llqrql I D.!.
6.2. Data import
The data import page allows the user to upload a file in .csv format consisting of data for the
corresponding year or month in the supplied template (see Figure 3). The imported data is checked
for correctness of categorical data such as states, generation sources and consuming sectors. lf these
checks fail, the user is provided a report of the reasons for failure, based on which the user can re-
import after correcting the data or adding new categories in the'Masters' page. ln addition, there
are configurable checks for the actual parametric data that is being imported, such as installed
capacity, quantity produced and quantity consumed. lf these checks fail, a warning is raised which
can either be overridden by the user. lf the data is determined to be wrong, it can be corrected and
re-imported.
The data that is imported can either be marked "Provisional" or "Final". "Provisional" data can be
overwritten multiple times, but "Final" data cannot be overwritten. The import of both "Provisional"
and "Final" data needs to be approved by a user with suitable privileges.
Hose Search Frgure 3: Datd import pdge
l{6t s lLr. tr.r! Domldd fbs APP
' D.t S.Lm.
caregory : c.nqrnption
:
?oli -
(,r,""r li'p,",i,r."a
o.o,se. , N! 6L el.<t.c.
6.3. Import log
The history of data imported to the database can be seen on the 'lmport Log' page. This page
provides a search facility to narrow down the history to specific sector and sub-sector. The search
results indicate the period for which data was imported, when the data was imported, whether it is
final or provisional, and which user imported the data (refer to Figure 4).
Figure 4: lmport Log
i'&,r '.l:::" r,ta.r'B n .t D.1, D.mr6d Frow App o'ra u*i
Drr. S!1.8. AdBin
El€d,,civ tklkiiyr@drh ;:
- !3n:?rek , L.nr,rorth
Ele.(i.ity aapelry f!..$ry;
Us Y.d Month 6n l hpon Dar.
2019
6,4. Fetching data from live government portals
As mentioned in Section 3.2, data from live government portals has been added to the Dashboards
This data is accessed through the APls that have been provided. The trigger for fetching the data is
provided through the workflow system under the tab 'APP Data' (see Figure 5). When the portal is
chosen and'Fetch Data' button is clicked, the data is fetched from the respective portal and added
to the database. This data is then available for download through the data download interface
described in Section 7. Figure 5: Fetching live portal data
Stafus
Output Type
ApiURLl
ApiURL2
t 5/01/2020 1al8:31
Data Processed Suc(essfully on 2020{1-'15 16118130
http/101.203 13a 1 tg,/Ujala .Vap,/.ommorlc€tNationaDarar'{D)
httpi//103.203.1 18 139,4riala_.Vapilommon/&tstateData{StatelD}?applian€etype={lD}
lD=2 rneans 8uib, lD=l means Fan, lD = 4 means Tube lighl
6.5. Generating the Energy flow [Sankey) and Energy balance tables
One of the im portant featu res of the Dashboards is to provide energy flow (Sankey) diagrams and
energy balance tables for the years for which final data is available. The workflow system provides
the ability to inspect if all yearly data is available for a particular year, and if the data exists, allow for
the energy flow diagram and energy balance table to be generated using the data. Figure 6 has a
snapshot of the same.
Figure 5: Generation of energy flow Sankey diagram and energy balance tables
'lmport lmpon Los Mr!L.3 lrcla D.l! O(MlsdAPP o'I. U3..
Adn'n
.
tlementr lor whi(h dnla is impo.tei {or This year
Y.a .rE D.lli*
Corl P.adudion Dra.ni(
aal rmp.r1 Co.ruoi.g scd.'
aelJlrrdu.li.rn Doin.stk St l€ .nd aompary wis.
(ft| rnrlr.,r!E9. !r.tor*r.
.Gl.on\ump!io. tr.re a.d ueag. sd.r wi{
aoal hr,.n an{.t.y 16.
Auxiliary cctrrlnptD. d.b
6,6, User administration -
'User admin' page
Accounts in the workflow system are assigned one of four roles -
user, approver, user admin and
developer admin - with a hierarchical set of privileges. lJser accou nts can upload data while the
approver accounts can approve the uploaded data and make changes to the master tables in
addition to uploading data- The user admin role can add, change or remove user accounts. The
developer admin account can undertake certain maintenance and development tasks such as
updating the meta-data and queries for data download. An 'Energy Data Portal and Application
Overview' document shall be provided at the time of deployment. This document consists details of
privileges accorded to each of the above roles.
t
Gln€iatoElrlancoDblii ..,tttu
2016 7. Improved data download
The first version of the Dashboards provides a way to download data in csv format through the
"Download" button at the bottom right corner provided by the Tableau@ data visualisation software
with which the charts are created. Data thus downloaded is not always clean, in that it may consist
of intermediate data used to produce the visualisations within Tableau. ln addition, this download
facility does not provide the user the ability to choose the category of data in line with the structure
of the Dashboards. As part of the current enhancements, a cleaner, more intuitive way of
downloading data from the portal is provided.
The improved data download interface can be accessed by NlTl Aayog staff from the workflow
system and by the general public through the lndia Energy Dashboards portal
{the blue button seen
in Figure 7). The user can choose the sector and category of data as organised on the lndia Energy
Dashboards. When the 'Download' button is pressed, the data is downloaded in a csv format. The
user can choose the level of detail by selecting either aggregate or itemised detail (see Figure 8).
The public version of the data download application will be enabled on the Dashboards when the
necessary lT security clearances are received from N lC.
Figure 7: Download data button on the Dashboards (top right corner, next to the help button)
NITIAayog INDIA' ''
,DASHBOARDS 8IILIS.r[
I'IDIA ETiERGY
Electricityconsumpton OOO
\+
tigure 8: Data download application interface
TLECTfu'ITYEl.lrrL. tr f,l.r:ll,l,/ CeBBna
g A!.ilahiil! !t Po*{
(!'
&e-e"t
(l
ncoi,"a
B. Feedback forum
The energy data user community could be engaged to help improve the quality of the data and
interface of the Dashboards. A feedback forum has been added to the Dashboards in order to
facilitate feedback from data users (see Figure 9). Figure 9: Feedback forum interface on the
Feedback & Sugqestlons topic: Overvi€w
0 Colnmenls lnda, Energy Dashbo.rd (ednrl 3 Oisqrs' Pnva€y Policy
ar Eecffifteni' tr6 IG@
'E nergy Balance'page
5 t-ogi, -
Stan the dis.russi.n. .
""
s'cn * yrtx o.so.,s @
i*(}#@
The feedback forum consists of separate sub-forums for each domain listed in the left hand side
menu on the Dashboards - Overview, Electricitv, Coal, Oil, Gas, State and References. This level of
granularity reduces clutter so that the feedback is relevant for the domain being viewed while also
ensuring that feedback is easier to track.
The feedback forum will be moderated by a respected energy sector researcher/analyst identified by
NlTl Aayog. Dr Rangan Baner.jee, Professor, llT Bombay, has agreed to moderate the forum.
9. Deployment and support
The enhanced Dashboards and the workflow application was deployed on NlTl Aayog servers and is
publicly accessible through the web at http://niti.sov.in/edm. The Dashboards were launched by the
Vice Chairperson, NlTl Aayog on Aprtl12'6 2O2!. The undedying database and Tableauo dashboards
were handed over to NlTl Aayog and NIC staif for future maintenance and feature addition. Training
was provided to NlTl Aayog and NIC staff at the time of hand over, and the various features were
explained along with a walk through of a typical workflow process.
Prayas (Energy Group) assisted NlTl Aayog staff over phone/audio conferences on an as needed
basis from May 2020 to April 2021. During this period, recent years' data has been incorporated into
the Dashboards by NlTl Aayog and data is available up to 2019-20, and in some cases up to 2O2O-21.
10. Future work
NlTl Aayog should institutionalise the data update activity so that the Dashboards continues to
remain relevant by having the latest available data.
ln addition, there is scope for significantly enhancing the amount of data and visualisations available
through the Dashboards. Following are some suggested areas of future work.
Regulatory data can be expanded to more states and DISCOMs using the provided formats
More live portals can be added by requesting for public Apls
The energy data user community can be engaged and based on the feedback received
o any data inconsistencies can be fixed and
o new data and dashboards can be added to lndia Energy Dashboards.
NlTl Aayog can consider a quarterly newsletter in which updates made to lndia Energy
Dash boards are commu nicated.