<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Promoting financial products to bank customers
In 2016, a retail bank sold several products (mortgage account, savings account, and pension account) to its customers.
It kept a record of all historical data, and this data is available for analysis and reuse.
Following a merger in 2017, the bank has new customers and wants to start some marketing campaigns. 
The budget for the campaigns is limited. The bank wants to contact a customer and propose only one product.
The marketing department needs to decide
Step1:  Understand the historical data
Load 2016 historical data, analyze it visually, and train a classifier to predict 2017 sales.
Load the historical customer data with their purchases (Mortgage, Savings, and Pension).
Step2:  Check the 2016 customers
Step3:  Do some visual analysis of the historical data
It's possible to use pandas plotting capabilities, but it would require a new version of it. This Notebook relies on matplotlib as it is present everywhere.
Step4:  In the following visualization, you can see the behavior of the 2016 customers for the three products.
The green color indicates that a customer bought a product; red indicates a customer did not buy a product.
The depth of the color indicates the number of purchases or non-purchases.
Step5:  Understanding the 2016 customers
We can see that
Step6:  Let's use the following columns as machine-learning features
Step7:  We use a standard basic support gradient boosting algorithm to predict whether a customer might by product A, B, or C.
Step8:  New customer data and predictions
Load new customer data, predict behaviors using trained classifier, and do some visual analysis.
We have all the characteristics of the new customers, as for the 2016 clients, but the new customers did not buy any product yet.
Load new customer data
Step9:  Predict behaviors of the new customers
Step10:  Package new data with predictions for optimization
Step11:  Do some visual analysis of the predicted data
Step12:  The predicted data has the same semantic as the base data, with even more clear frontiers
Step13:  Remarks on the prediction
The goal is to contact the customers to sell them only one product, so we cannot select all of them.
This increases the complexity of the problem
Step14:  Get business decisions on the 2017 data
Assign campaigns to customers
We have predicted who will buy what in the list of new customers.
However, we do not have the budget to contact all of them. We have various contact channels with different costs and effectiveness.
Furthermore, if we contact somebody, we don't want to frustrate them by proposing multiple products; we want to propose only one product per customer.
Some input data for optimization
Step15:  Using a greedy algorithm
We create a custom algorithm that ensures 10% of offers are made per channel by choosing the most promising per channel. The algorithm then continues to add offers until the budget is reached.
Step16:  The greedy algorithm only gives a revenue of \$50.8K.
Using IBM Decision Optimization CPLEX Modeling for Python
Let's create the optimization model to select the best ways to contact customers and stay within the limited budget.
Step17:  Set up the prescriptive model
There are two ways to solve the model
Step18:  docplex solve methods take various arguments
Step19:  Define the decision variables
The integer decision variables channelVars, represent whether or not a customer will be made an offer for a particular product via a particular channel.
The integer decision variable totaloffers represents the total number of offers made.
The continuous variable budgetSpent represents the total cost of the offers made.
Step20:  Set up the constraints
Offer only one product per customer.
Compute the budget and set a maximum on it.
Compute the number of offers to be made.
Ensure at least 10% of offers are made via each channel.
Step21:  Express the objective
We want to maximize expected revenue, so we take into account the predicted behavior of each customer for each product.
Step22:  Solve with the Decision Optimization solve service
Step23:  Analyze the solution
Step24:  With the mathematical optimization, we made a better selection of customers.
What if our budget is increased?
If our manager is prepared to increase the allocated budget, they might want to know whether the additional budget campaigns would bring more revenue.
Step25:  The following cell takes a relatively long term to run because the jobs are run sequentially. The standard subscriptions to DOcplexcloud solve service only allow one job at a time, but you can buy special subscriptions with parallel solves. If you have such a subscription, modify the following cell to benefit from it.
Step26:  Due to the business constraints, we can address a maximum of 1680 customers with a \$35615 budget.
Any funds available above that amount won't be spent.
The expected revenue is \$87.1K.
Dealing with infeasibility
What about a context where we are in tight financial conditions, and our budget is very low?
We need to determine the minimum amount of budget needed to adress 1/20 of our customers.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
%matplotlib inline
Explanation: Promoting financial products to bank customers
In 2016, a retail bank sold several products (mortgage account, savings account, and pension account) to its customers.
It kept a record of all historical data, and this data is available for analysis and reuse.
Following a merger in 2017, the bank has new customers and wants to start some marketing campaigns. 
The budget for the campaigns is limited. The bank wants to contact a customer and propose only one product.
The marketing department needs to decide:
   * Who should be contacted?
   * Which product should be proposed? Proposing too many products is counter productive, so only one product per customer contact.
   * How will a customer be contacted? There are different ways, with different costs and efficiency.
   * How can they optimally use the limited budget?
   * Will such campaigns be profitable?
Predictive and prescriptive workflow
From the historical data, we can train a machine learning product-based classifier on customer profile (age, income, account level, ...) to predict whether a customer would subscribe to a mortgage, savings, or pension account.
* We can apply this predictive model to the new customers data to predict for each new customer what they will buy.
* On this new data, we decide which offers are proposed. Which product is offered to which customer through which channel:
   * a. with a greedy algorithm that reproduces what a human being would do
   * b. using an optimization model wih IBM Decision Optimization.
* The solutions can be displayed, compared, and analyzed.
<b>Table of contents</b>:
Understand the historical data
Predict the 2017 customer behavior
Get business decisions on the 2017 data
Conclusion on the decision making
This notebook requires a mathematical background.
<br/>If you're new to optimization, following the online free Decision Optimization tutorials (here and here) might help you get a better understanding of Mathematical Optimization.
This notebook is part of Prescriptive Analytics for Python
It requires either an installation of CPLEX Optimizers or it can be run on IBM Watson Studio Cloud (Sign up for a free IBM Cloud account
and you can start using Watson Studio Cloud right away).
The purpose of this Notebook is not to provide a perfect machine learning model nor a perfect optimization model.
The purpose is to show how easy it is to mix machine learning and CPLEX data transformations by doing a forecast, then getting fast and reliable decisions on this new data. 
This notebook takes some time to run because multiple optimization models are solved and compared in the part dedicated to what-if analysis. The time it takes depends on your subscription type, which determines what optimization service configuration is used.
How  decision optimization can help
Prescriptive analytics (decision optimization) technology recommends actions that are based on desired outcomes.  It takes into account specific scenarios, resources, and knowledge of past and current events. With this insight, your organization can make better decisions and have greater control of business outcomes.  
Prescriptive analytics is the next step on the path to insight-based actions. It creates value through synergy with predictive analytics, which analyzes data to predict future outcomes.  
Prescriptive analytics takes that prediction to the next level by suggesting the optimal way to handle that future situation. Organizations gain a strong competitive advantage by acting quickly in dynamic conditions and making superior decisions in uncertain environments.
<br/>
<u>With prescriptive analytics, you can:</u> 
Automate complex decisions and trade-offs to better manage your limited resources.
Take advantage of a future opportunity or mitigate a future risk.
Proactively update recommendations based on changing events.
Meet operational goals, increase customer loyalty, prevent threats and fraud, and optimize business processes.
End of explanation
known_behaviors = pd.read_csv("https://raw.githubusercontent.com/vberaudi/utwt/master/known_behaviors2.csv")
known_behaviors.head()
Explanation: Understand the historical data
Load 2016 historical data, analyze it visually, and train a classifier to predict 2017 sales.
Load the historical customer data with their purchases (Mortgage, Savings, and Pension).
End of explanation
a = known_behaviors[known_behaviors.Mortgage == 1]
b = known_behaviors[known_behaviors.Pension == 1]
c = known_behaviors[known_behaviors.Savings == 1]
print("Number of clients: %d" %len(known_behaviors))
print("Number of clients predicted to buy mortgage accounts: %d" %len(a))
print("Number of clients predicted to buy pension accounts: %d" %len(b))
print("Number of clients predicted to buy savings accounts: %d" %len(c))
known_behaviors["nb_products"] = known_behaviors.Mortgage + known_behaviors.Pension + known_behaviors.Savings
abc = known_behaviors[known_behaviors.nb_products > 1]
print("We have %d clients who bought several products" %len(abc))
abc = known_behaviors[known_behaviors.nb_products == 3]
print("We have %d clients who bought all the products" %len(abc))
products = ["Savings", "Mortgage", "Pension"]
Explanation: Check the 2016 customers
End of explanation
def plot_cloud_points(df):
    figure = plt.figure(figsize=(20, 5))
    my_cm  = ListedColormap(['#bb0000', '#00FF00'])
    axes = {p : ('age', 'income') if p != "Mortgage"else ('members_in_household', 'loan_accounts') for p in products}
    for product in products:
        ax = plt.subplot(1, len(products), products.index(product)+1)
        ax.set_title(product)
        axe = axes[product]
        plt.xlabel(axe[0])
        plt.ylabel(axe[1])
        ax.scatter(df[axe[0]], df[axe[1]], c=df[product], cmap=my_cm, alpha=0.5)
Explanation: Do some visual analysis of the historical data
It's possible to use pandas plotting capabilities, but it would require a new version of it. This Notebook relies on matplotlib as it is present everywhere.
End of explanation
plot_cloud_points(known_behaviors)
Explanation: In the following visualization, you can see the behavior of the 2016 customers for the three products.
The green color indicates that a customer bought a product; red indicates a customer did not buy a product.
The depth of the color indicates the number of purchases or non-purchases.
End of explanation
known_behaviors.columns
Explanation: Understanding the 2016 customers
We can see that:
   * The greater a customer's income, the more likely it is s/he will buy a savings account.
   * The older a customer is, the more likely it is s/he will buy a pension account.
   * There is a correlation between the number of people in a customer's household, the number of loan accounts held by the customer, and the likelihood a customer buys a mortgage account. To see the correlation, look at the upper right and lower left corners of the mortgage chart.
Predict the 2017 customer behavior
Create and train a simple machine-learning algorithm to predict what the new clients will buy.
End of explanation
cols = ['age', 'income', 'members_in_household', 'loan_accounts']
X = known_behaviors[cols]
ys = [known_behaviors[p] for p in products]
X.head()
Explanation: Let's use the following columns as machine-learning features:
End of explanation
from sklearn import svm
from sklearn import ensemble
classifiers = []
for i,p in enumerate(products):
    clf = ensemble.GradientBoostingClassifier()
    clf.fit(X, ys[i])
    classifiers.append(clf)
Explanation: We use a standard basic support gradient boosting algorithm to predict whether a customer might by product A, B, or C.
End of explanation
unknown_behaviors = pd.read_csv("https://raw.githubusercontent.com/vberaudi/utwt/master/unknown_behaviors.csv")
for c in unknown_behaviors.columns:
    assert c in known_behaviors.columns
to_predict = unknown_behaviors[cols]
print("Number of new customers: %d" %len(unknown_behaviors))
Explanation: New customer data and predictions
Load new customer data, predict behaviors using trained classifier, and do some visual analysis.
We have all the characteristics of the new customers, as for the 2016 clients, but the new customers did not buy any product yet.
Load new customer data
End of explanation
import warnings
warnings.filterwarnings('ignore')
predicted = [classifiers[i].predict(to_predict) for i in range(len(products))]
for i,p in enumerate(products):
    to_predict[p] = predicted[i]
to_predict["id"] = unknown_behaviors["customer_id"]
Explanation: Predict behaviors of the new customers
End of explanation
offers = to_predict
offers.head()
Explanation: Package new data with predictions for optimization
End of explanation
plot_cloud_points(offers)
Explanation: Do some visual analysis of the predicted data
End of explanation
a = offers[offers.Mortgage == 1]
b = offers[offers.Pension == 1]
c = offers[offers.Savings == 1]
print("Number of new customers: %d" %len(offers))
print("Number of customers predicted to buy mortgages: %d" %len(a))
print("Number of customers predicted to buy pensions: %d" %len(b))
print("Number of customers predicted to buy savings: %d" %len(c))
to_predict["nb_products"] = to_predict.Mortgage + to_predict.Pension + to_predict.Savings
abc = to_predict[to_predict.nb_products > 1]
print("We predicted that %d clients would buy more than one product" %len(abc))
abc = to_predict[to_predict.nb_products == 3]
print("We predicted that %d clients would buy all three products" %len(abc))
Explanation: The predicted data has the same semantic as the base data, with even more clear frontiers:
* for savings, there is a  clear frontier at $50K revenue.
* for pension, there is a clear frontier at 55 years old customers.
The training data contains customers who bought more than one product, let's see our prediction
End of explanation
offers.reset_index(inplace=True)
Explanation: Remarks on the prediction
The goal is to contact the customers to sell them only one product, so we cannot select all of them.
This increases the complexity of the problem: we need to determine the best contact channel, but also need to select which product will be sold to a given customer.
It may be hard to compute this. In order to check, we will use two techniques:
   * a greedy algorithm
   * CPLEX, the IBM leading optimization solver.
End of explanation
# How much revenue is earned when selling each product
productValue = [200, 300, 400]
value_per_product = {products[i] : productValue[i] for i in range(len(products))}
# Total available budget
availableBudget = 25000
# For each channel, cost of making a marketing action and success factor
channels =  pd.DataFrame(data=[("gift", 20.0, 0.20), 
                               ("newsletter", 15.0, 0.05), 
                               ("seminar", 23.0, 0.30)], columns=["name", "cost", "factor"])
offersR = range(0, len(offers))
productsR = range(0, len(products))
channelsR = range(0, len(channels))
Explanation: Get business decisions on the 2017 data
Assign campaigns to customers
We have predicted who will buy what in the list of new customers.
However, we do not have the budget to contact all of them. We have various contact channels with different costs and effectiveness.
Furthermore, if we contact somebody, we don't want to frustrate them by proposing multiple products; we want to propose only one product per customer.
Some input data for optimization
End of explanation
gsol = pd.DataFrame()
gsol['id'] = offers['id']
budget = 0
revenue = 0
for product in products:
    gsol[product] = 0
   
noffers = len(offers)
# ensure the 10% per channel by choosing the most promising per channel
for c in channelsR: #, channel in channels.iterrows():
    i = 0;
    while (i< ( noffers // 10 ) ):
        # find a possible offer in this channel for a customer not yet done
        added = False
        for o  in offersR:
            already = False
            for product in products:   
                if gsol.get_value(index=o, col=product) == 1:
                    already = True
                    break
            if already:
                continue
            possible = False
            possibleProduct = None
            for product in products:
                if offers.get_value(index=o, col=product) == 1:
                    possible = True
                    possibleProduct = product
                    break
            if not possible:
                continue
            #print "Assigning customer ", offers.get_value(index=o, col="id"), " with product ", product, " and channel ", channel['name']
            gsol.set_value(index=o, col=possibleProduct, value=1)
            i = i+1
            added = True
            budget = budget + channels.get_value(index=c, col="cost")
            revenue = revenue + channels.get_value(index=c, col="factor")*value_per_product[product]            
            break
        if not added:
            print("NOT FEASIBLE")
            break
# add more to complete budget       
while (True):
    added = False
    for c, channel in channels.iterrows():
        if (budget + channel.cost > availableBudget):
            continue
        # find a possible offer in this channel for a customer not yet done
        for o  in offersR:
            already = False
            for product in products:   
                if gsol.get_value(index=o, col=product) == 1:
                    already = True
                    break
            if already:
                continue
            possible = False
            possibleProduct = None
            for product in products:
                if offers.get_value(index=o, col=product) == 1:
                    possible = True
                    possibleProduct = product
                    break
            if not possible:
                continue
            #print "Assigning customer ", offers.get_value(index=o, col="id"), " with product ", product, " and channel ", channel['name']
            gsol.set_value(index=o, col=possibleProduct, value=1)
            i = i+1
            added = True
            budget = budget + channel.cost
            revenue = revenue + channel.factor*value_per_product[product]            
            break
    if not added:
        print("FINISH BUDGET")
        break
    
print(gsol.head())
a = gsol[gsol.Mortgage == 1]
b = gsol[gsol.Pension == 1]
c = gsol[gsol.Savings == 1]
abc = gsol[(gsol.Mortgage == 1) | (gsol.Pension == 1) | (gsol.Savings == 1)]
print("Number of clients: %d" %len(abc))
print("Numbers of Mortgage offers: %d" %len(a))
print("Numbers of Pension offers: %d" %len(b))
print("Numbers of Savings offers: %d" %len(c))
print("Total Budget Spent: %d" %budget)
print("Total revenue: %d" %revenue)
comp1_df = pd.DataFrame(data=[["Greedy", revenue, len(abc), len(a), len(b), len(c), budget]], columns=["Algorithm","Revenue","Number of clients","Mortgage offers","Pension offers","Savings offers","Budget Spent"])
Explanation: Using a greedy algorithm
We create a custom algorithm that ensures 10% of offers are made per channel by choosing the most promising per channel. The algorithm then continues to add offers until the budget is reached.
End of explanation
import sys
import docplex.mp
Explanation: The greedy algorithm only gives a revenue of \$50.8K.
Using IBM Decision Optimization CPLEX Modeling for Python
Let's create the optimization model to select the best ways to contact customers and stay within the limited budget.
End of explanation
url = None
key = None
Explanation: Set up the prescriptive model
There are two ways to solve the model:
* Subscribe to our private cloud offer or Decision Optimization on Cloud solve service here.
* Use a local solver, a licensed installation of CPLEX Optimization Studio. Note that this model is too big to solve using the Community Edition, so if url and key are None, the script will fail.
End of explanation
from docplex.mp.context import Context
context = Context.make_default_context()
context.solver.docloud.url = url
context.solver.docloud.key = key
context.solver.agent = 'docloud'
from docplex.mp.model import Model
mdl = Model(name="marketing_campaign", checker='on', context=context)
Explanation: docplex solve methods take various arguments: you can pass the api and key at each function call or you can put them at model declaration with a context.
End of explanation
channelVars = mdl.binary_var_cube(offersR, productsR, channelsR)
Explanation: Define the decision variables
The integer decision variables channelVars, represent whether or not a customer will be made an offer for a particular product via a particular channel.
The integer decision variable totaloffers represents the total number of offers made.
The continuous variable budgetSpent represents the total cost of the offers made.
End of explanation
# At most 1 product is offered to each customer
mdl.add_constraints( mdl.sum(channelVars[o,p,c] for p in productsR for c in channelsR) <=1
                   for o in offersR)
# Do not exceed the budget
mdl.add_constraint( mdl.sum(channelVars[o,p,c]*channels.get_value(index=c, col="cost") 
                                           for o in offersR 
                                           for p in productsR 
                                           for c in channelsR)  <= availableBudget, "budget")  
# At least 10% offers per channel
for c in channelsR:
    mdl.add_constraint(mdl.sum(channelVars[o,p,c] for p in productsR for o in offersR) >= len(offers) // 10)
mdl.print_information()
Explanation: Set up the constraints
Offer only one product per customer.
Compute the budget and set a maximum on it.
Compute the number of offers to be made.
Ensure at least 10% of offers are made via each channel.
End of explanation
obj = 0
for c in channelsR:
    for p in productsR:
        product=products[p]
        coef = channels.get_value(index=c, col="factor") * value_per_product[product]
        obj += mdl.sum(channelVars[o,p,c] * coef* offers.get_value(index=o, col=product) for o in offersR)
mdl.maximize(obj)
Explanation: Express the objective
We want to maximize expected revenue, so we take into account the predicted behavior of each customer for each product.
End of explanation
mdl.parameters.timelimit = 30
s = mdl.solve()
assert s, "No Solution !!!"
print(mdl.get_solve_status())
print(mdl.get_solve_details())
Explanation: Solve with the Decision Optimization solve service
End of explanation
totaloffers = mdl.sum(channelVars[o,p,c] 
                      for o in offersR
                      for p in productsR 
                      for c in channelsR)
mdl.add_kpi(totaloffers, "nb_offers")
budgetSpent = mdl.sum(channelVars[o,p,c]*channels.get_value(index=c, col="cost") 
                                           for o in offersR 
                                           for p in productsR 
                                           for c in channelsR)
mdl.add_kpi(budgetSpent, "budgetSpent")
for c in channelsR:
    channel = channels.get_value(index=c, col="name")
    kpi = mdl.sum(channelVars[o,p,c] for p in productsR for o in offersR)
    mdl.add_kpi(kpi, channel)
for p in productsR:
    product = products[p]
    kpi = mdl.sum(channelVars[o,p,c] for c in channelsR for o in offersR)
    mdl.add_kpi(kpi, product)
mdl.report()
comp2_df = pd.DataFrame(data=[["CPLEX", mdl.objective_value, mdl.kpi_value_by_name('nb_offers'), mdl.kpi_value_by_name('Mortgage'), mdl.kpi_value_by_name('Pension'), mdl.kpi_value_by_name('Savings'), mdl.kpi_value_by_name('budgetSpent')]], columns=["Algorithm","Revenue","Number of clients","Mortgage offers","Pension offers","Savings offers","Budget Spent"])
comp_df = comp1_df.append(comp2_df, ignore_index=True)
comp_df
comp_df.set_index("Algorithm", inplace=True)
    
my_plot = comp_df['Revenue'].plot(kind='bar')
Explanation: Analyze the solution
End of explanation
#get the hand on the budget constraint
ct = mdl.get_constraint_by_name("budget")
Explanation: With the mathematical optimization, we made a better selection of customers.
What if our budget is increased?
If our manager is prepared to increase the allocated budget, they might want to know whether the additional budget campaigns would bring more revenue.
End of explanation
res = []
for i in range(20):
    ct.rhs = availableBudget+1000*i
    s = mdl.solve()
    assert s, "No Solution !!!"
    res.append((availableBudget+1000*i, mdl.objective_value, mdl.kpi_value_by_name("nb_offers"), mdl.kpi_value_by_name("budgetSpent")))
mdl.report()
pd.DataFrame(res, columns=["budget", "revenue", "nb_offers", "budgetSpent"])
Explanation: The following cell takes a relatively long term to run because the jobs are run sequentially. The standard subscriptions to DOcplexcloud solve service only allow one job at a time, but you can buy special subscriptions with parallel solves. If you have such a subscription, modify the following cell to benefit from it.
End of explanation
ct.rhs = 0
s = mdl.solve()
if not s:
    #rename the constraint with a "low" prefix to automatically put a low priority on it.
    ct.name = "low_budget"
    #setting all bool vars to 0 is an easy relaxation, so let's refuse it and force to offer something to 1/3 of the clients
    mdl.add_constraint(totaloffers >= len(offers)//20, ctname="high")
    
    # solve has failed, we try relaxation, based on constraint names
    # constraints are prioritized according to their names
    # if a name contains "low", it has priority LOW
    # if a ct name contains "medium" it has priority MEDIUM
    # same for HIGH
    # if a constraint has no name or does not match any, it is not relaxable.
    from docplex.mp.relaxer import Relaxer
    relaxer = Relaxer(prioritizer='match', verbose=True)
    relaxed_sol = relaxer.relax(mdl)
    relaxed_ok = relaxed_sol is not None
    assert relaxed_ok, "relaxation failed"
    relaxer.print_information()
mdl.report()
print(mdl.get_solve_status())
print(mdl.get_solve_details())
Explanation: Due to the business constraints, we can address a maximum of 1680 customers with a \$35615 budget.
Any funds available above that amount won't be spent.
The expected revenue is \$87.1K.
Dealing with infeasibility
What about a context where we are in tight financial conditions, and our budget is very low?
We need to determine the minimum amount of budget needed to adress 1/20 of our customers.
End of explanation
<END_TASK>