<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Applying refutation tests to the Lalonde and IHDP datasets
Import the Dependencies
Step1:  Loading the Dataset
Infant Health and Development Program Dataset (IHDP)
The measurements used are on the child—birth weight, head circumference, weeks bornpreterm, birth order, ﬁrst born, neonatal health index (see Scott and Bauer 1989), sex, twinstatus—as well as behaviors engaged in during the pregnancy—smoked cigarettes, drankalcohol, took drugs—and measurements on the mother at the time she gave birth—age,marital status, educational attainment (did not graduate from high school, graduated fromhigh school, attended some college but did not graduate, graduated from college), whethershe worked during pregnancy, whether she received prenatal care—and the site (8 total) inwhich the family resided at the start of the intervention. There are 6 continuous covariatesand 19 binary covariates.
Reference
Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1), 217-240. https
Step2:  Lalonde Dataset
A data frame with 445 observations on the following 12 variables.
age
Step3:  Step 1
Step4:  Lalonde
Step5:  Step 2
Step6:  Lalonde
Step7:  Step 3
Step8:  Lalonde
Step9:  Step 4
Step10:  Replace Treatment with Placebo
Step11:  Remove Random Subset of Data
Step12:  Lalonde
Add Random Common Cause
Step13:  Replace Treatment with Placebo
Step14:  Remove Random Subset of Data
<ASSISTANT_TASK:>
Python Code:
import dowhy
from dowhy import CausalModel
import pandas as pd
import numpy as np
# Config dict to set the logging level
import logging.config
DEFAULT_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'loggers': {
        '': {
            'level': 'WARN',
        },
    }
}
logging.config.dictConfig(DEFAULT_LOGGING)
# Disabling warnings output
import warnings
from sklearn.exceptions import DataConversionWarning
warnings.filterwarnings(action='ignore', category=DataConversionWarning)
Explanation: Applying refutation tests to the Lalonde and IHDP datasets
Import the Dependencies
End of explanation
data = pd.read_csv("https://raw.githubusercontent.com/AMLab-Amsterdam/CEVAE/master/datasets/IHDP/csv/ihdp_npci_1.csv", header = None)
col =  ["treatment", "y_factual", "y_cfactual", "mu0", "mu1" ,]
for i in range(1,26):
    col.append("x"+str(i))
data.columns = col
data = data.astype({"treatment":'bool'}, copy=False)
data.head()
Explanation: Loading the Dataset
Infant Health and Development Program Dataset (IHDP)
The measurements used are on the child—birth weight, head circumference, weeks bornpreterm, birth order, ﬁrst born, neonatal health index (see Scott and Bauer 1989), sex, twinstatus—as well as behaviors engaged in during the pregnancy—smoked cigarettes, drankalcohol, took drugs—and measurements on the mother at the time she gave birth—age,marital status, educational attainment (did not graduate from high school, graduated fromhigh school, attended some college but did not graduate, graduated from college), whethershe worked during pregnancy, whether she received prenatal care—and the site (8 total) inwhich the family resided at the start of the intervention. There are 6 continuous covariatesand 19 binary covariates.
Reference
Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1), 217-240. https://doi.org/10.1198/jcgs.2010.08162
End of explanation
from rpy2.robjects import r as R
from os.path import expanduser
home = expanduser("~")
%reload_ext rpy2.ipython
# %R install.packages("Matching")
%R library(Matching)
%R data(lalonde)
%R -o lalonde
lalonde = lalonde.astype({'treat':'bool'}, copy=False)
lalonde.head()
Explanation: Lalonde Dataset
A data frame with 445 observations on the following 12 variables.
age:
age in years.
educ:
years of schooling.
black:
indicator variable for blacks.
hisp:
indicator variable for Hispanics.
married:
indicator variable for martial status.
nodegr:
indicator variable for high school diploma.
re74:
real earnings in 1974.
re75:
real earnings in 1975.
re78:
real earnings in 1978.
u74:
indicator variable for earnings in 1974 being zero.
u75:
indicator variable for earnings in 1975 being zero.
treat:
an indicator variable for treatment status.
References
Dehejia, Rajeev and Sadek Wahba. 1999.``Causal Effects in Non-Experimental Studies: Re-Evaluating the Evaluation of Training Programs.'' Journal of the American Statistical Association 94 (448): 1053-1062.
LaLonde, Robert. 1986. ``Evaluating the Econometric Evaluations of Training Programs.'' American Economic Review 76:604-620.
End of explanation
# Create a causal model from the data and given common causes
common_causes = []
for i in range(1, 26):
    common_causes += ["x"+str(i)]
ihdp_model = CausalModel(
                data=data,
                treatment='treatment',
                outcome='y_factual',
                common_causes=common_causes
            )
ihdp_model.view_model(layout="dot")
from IPython.display import Image, display
display(Image(filename="causal_model.png"))
Explanation: Step 1: Building the model
IHDP
End of explanation
lalonde_model = CausalModel(
                data=lalonde,
                treatment='treat',
                outcome='re78',
                common_causes='nodegr+black+hisp+age+educ+married'.split('+')
            )
lalonde_model.view_model(layout="dot")
from IPython.display import Image, display
display(Image(filename="causal_model.png"))
Explanation: Lalonde
End of explanation
#Identify the causal effect for the ihdp dataset
ihdp_identified_estimand = ihdp_model.identify_effect(proceed_when_unidentifiable=True)
print(ihdp_identified_estimand)
Explanation: Step 2: Identification
IHDP
End of explanation
#Identify the causal effect for the lalonde dataset
lalonde_identified_estimand = lalonde_model.identify_effect(proceed_when_unidentifiable=True)
print(lalonde_identified_estimand)
Explanation: Lalonde
End of explanation
ihdp_estimate = ihdp_model.estimate_effect(
                    ihdp_identified_estimand,
                    method_name="backdoor.propensity_score_weighting"
                )
print("The Causal Estimate is " + str(ihdp_estimate.value))
Explanation: Step 3: Estimation (using propensity score weighting)
IHDP
End of explanation
lalonde_estimate = lalonde_model.estimate_effect(
                        lalonde_identified_estimand,
                        method_name="backdoor.propensity_score_weighting"
                    )
print("The Causal Estimate is " + str(lalonde_estimate.value))
Explanation: Lalonde
End of explanation
ihdp_refute_random_common_cause = ihdp_model.refute_estimate(
                                        ihdp_identified_estimand,
                                        ihdp_estimate,
                                        method_name="random_common_cause"
                                    )
print(ihdp_refute_random_common_cause)
Explanation: Step 4:  Refutation
IHDP
Add Random Common Cause
End of explanation
ihdp_refute_placebo_treatment = ihdp_model.refute_estimate(
                                    ihdp_identified_estimand,
                                    ihdp_estimate,
                                    method_name="placebo_treatment_refuter",
                                    placebo_type="permute"
                                )
print(ihdp_refute_placebo_treatment)
Explanation: Replace Treatment with Placebo
End of explanation
ihdp_refute_random_subset = ihdp_model.refute_estimate(
                                    ihdp_identified_estimand,
                                    ihdp_estimate,
                                    method_name="data_subset_refuter",
                                    subset_fraction=0.8
                            )
print(ihdp_refute_random_subset)
Explanation: Remove Random Subset of Data
End of explanation
lalonde_refute_random_common_cause = lalonde_model.refute_estimate(
                                            lalonde_identified_estimand,
                                            lalonde_estimate,
                                            method_name="random_common_cause"
                                        )
print(lalonde_refute_random_common_cause)
Explanation: Lalonde
Add Random Common Cause
End of explanation
lalonde_refute_placebo_treatment = lalonde_model.refute_estimate(
                                        lalonde_identified_estimand,
                                        lalonde_estimate,
                                        method_name="placebo_treatment_refuter",
                                        placebo_type="permute"
                                    )
print(lalonde_refute_placebo_treatment)
Explanation: Replace Treatment with Placebo
End of explanation
lalonde_refute_random_subset = lalonde_model.refute_estimate(
                                    lalonde_identified_estimand,
                                    lalonde_estimate,
                                    method_name="data_subset_refuter",
                                    subset_fraction=0.9
                                )
print(lalonde_refute_random_subset)
Explanation: Remove Random Subset of Data
End of explanation
<END_TASK>