<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Validation of gf_eia923
This notebook runs sanity checks on the Generation Fuel data that are reported in EIA Form 923. These are the same tests which are run by the gf_eia923 validation tests by PyTest.  The notebook and visualizations are meant to be used as a diagnostic tool, to help understand what's wrong when the PyTest based data validations fail for some reason.
Step1:  Get the original EIA 923 data
First we pull the original (post-ETL) EIA 923 data out of the database. We will use the values in this dataset as a baseline for checking that latter aggregated data and derived values remain valid.  We will also eyeball these values here to make sure they are within the expected range. This may take a minute or two depending on the speed of your machine.
Step2:  Validation Against Fixed Bounds
Some of the variables reported in this table have a fixed range of reasonable values, like the heat content per unit of a given fuel type.  These varaibles can be tested for validity against external standards directly.  In general we have two kinds of tests in this section
Step3:  Coal Heat Content
Step4:  Oil Heat Content
Step5:  Gas Heat Content
Step6:  Validate Monthly Aggregation
It's possible that the distribution will change as a function of aggregation, or we might make an error in the aggregation process. These tests check that a collection of quantiles for the original and the data aggregated by month have internally consistent values.
Step7:  Validate Annual Aggregation
It's possible that the distribution will change as a function of aggregation, or we might make an error in the aggregation process. These tests check that a collection of quantiles for the original and the data aggregated by year have internally consistent values.
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload
%autoreload 2
import sys
import pandas as pd
import sqlalchemy as sa
import pudl
import warnings
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(stream=sys.stdout)
formatter = logging.Formatter('%(message)s')
handler.setFormatter(formatter)
logger.handlers = [handler]
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
plt.style.use('ggplot')
mpl.rcParams['figure.figsize'] = (10,4)
mpl.rcParams['figure.dpi'] = 150
pd.options.display.max_columns = 56
pudl_settings = pudl.workspace.setup.get_defaults()
ferc1_engine = sa.create_engine(pudl_settings['ferc1_db'])
pudl_engine = sa.create_engine(pudl_settings['pudl_db'])
pudl_settings
Explanation: Validation of gf_eia923
This notebook runs sanity checks on the Generation Fuel data that are reported in EIA Form 923. These are the same tests which are run by the gf_eia923 validation tests by PyTest.  The notebook and visualizations are meant to be used as a diagnostic tool, to help understand what's wrong when the PyTest based data validations fail for some reason.
End of explanation
pudl_out_orig = pudl.output.pudltabl.PudlTabl(pudl_engine, freq=None)
gf_eia923_orig = pudl_out_orig.gf_eia923()
Explanation: Get the original EIA 923 data
First we pull the original (post-ETL) EIA 923 data out of the database. We will use the values in this dataset as a baseline for checking that latter aggregated data and derived values remain valid.  We will also eyeball these values here to make sure they are within the expected range. This may take a minute or two depending on the speed of your machine.
End of explanation
gf_eia923_orig.sample(10)
Explanation: Validation Against Fixed Bounds
Some of the variables reported in this table have a fixed range of reasonable values, like the heat content per unit of a given fuel type.  These varaibles can be tested for validity against external standards directly.  In general we have two kinds of tests in this section:
* Tails: are the exteme values too extreme? Typically, this is at the 5% and 95% level, but depending on the distribution, sometimes other thresholds are used.
* Middle: Is the central value of the distribution where it should be?
Fields that need checking:
These are all contained in the frc_eia923 table data validations, and those should just be re-used if possible.  Ugh, names not all the same though.  Annoying.
* fuel_mmbtu_per_unit (BIT, SUB, LIG, coal, DFO, oil, gas)
End of explanation
pudl.validate.plot_vs_bounds(gf_eia923_orig, pudl.validate.gf_eia923_coal_heat_content)
Explanation: Coal Heat Content
End of explanation
pudl.validate.plot_vs_bounds(gf_eia923_orig, pudl.validate.gf_eia923_oil_heat_content)
Explanation: Oil Heat Content
End of explanation
pudl.validate.plot_vs_bounds(gf_eia923_orig, pudl.validate.gf_eia923_gas_heat_content)
Explanation: Gas Heat Content
End of explanation
pudl_out_month = pudl.output.pudltabl.PudlTabl(pudl_engine, freq="MS")
gf_eia923_month = pudl_out_month.gf_eia923()
pudl.validate.plot_vs_agg(gf_eia923_orig, gf_eia923_month, pudl.validate.gf_eia923_agg)
Explanation: Validate Monthly Aggregation
It's possible that the distribution will change as a function of aggregation, or we might make an error in the aggregation process. These tests check that a collection of quantiles for the original and the data aggregated by month have internally consistent values.
End of explanation
pudl_out_year = pudl.output.pudltabl.PudlTabl(pudl_engine, freq="AS")
gf_eia923_year = pudl_out_year.gf_eia923()
pudl.validate.plot_vs_agg(gf_eia923_orig, gf_eia923_year, pudl.validate.gf_eia923_agg)
Explanation: Validate Annual Aggregation
It's possible that the distribution will change as a function of aggregation, or we might make an error in the aggregation process. These tests check that a collection of quantiles for the original and the data aggregated by year have internally consistent values.
End of explanation
<END_TASK>