<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 I download non-overlapping randomly selected samples from file auth.txt.gz so that each file contains roughly the same number of successes and fails.
Step1:  I here repeat my procedure for generating labeled data and features for training/test data.
Step2:  I just discovered that my sample sets do not contain the same number of features. 
Below I am checking to see what the source of this difference.
Step3:  This is potentially different spelling of two different commands/labels. For now I will just remove all the labels that are not present in 15 files of data I have just downloaded. If the scores for machine learning will change noticeably. I will look into ways to clean and incorporate this data.
Step4:  Machine learning with logistic regression with Lasso
<ASSISTANT_TASK:>
Python Code:
all_df=[]
nfiles=15
for i in range(nfiles):
    filename = 'msample%d.csv' % i
    print i
    all_df.append(pd.read_csv(filename, header=None))
all_df[0]
Y=[]
for i in range(nfiles):
    Y.append(all_df[i][8]=='Success')
Y[1]
Explanation: I download non-overlapping randomly selected samples from file auth.txt.gz so that each file contains roughly the same number of successes and fails.
End of explanation
def map_user(x):
    if x.startswith('C'):
        return 'C'
    elif x.startswith('U'):
        return 'U'
    else:
        return x
X=[]
for i in range(nfiles):
    df=all_df[i]
    df["source_user"], df["source_domain"] = zip(*df[1].str.split('@').tolist())
    df["source_user"]=df["source_user"].str.rstrip('$')
    df["destination_user"], df["destination_domain"] = zip(*df[2].str.split('@').tolist())
    df["destination_user"]=df["destination_user"].str.rstrip('$')
    df['source_class']=df['source_user'].map(map_user)
    df['destination_class']=df['destination_user'].map(map_user)
    x=pd.DataFrame.from_items([
    ('time', (df[0]%(24*60*60)).astype(int))])
    x['same_user']= (df['destination_user']==df['source_user'])
    x['same_domain']=(df['destination_domain']==df['source_domain'])
    x['source_user_comp_same']=(df[3]==df['source_user'])
    x['destination_user_comp_same']=(df['destination_user']==df[4])
    x['same_comp']=(df[3]==df[4])
    x['source_domain_comp_same']=(df[3]==df['source_domain'])
    x['destination_domain_comp_same']=(df['destination_domain']==df[4])
    
    for j in [5,6, 7]:
        for label in sorted(df[j].unique()):
            if label=='?':
                if j==5:
                    x['?_authentication type']=(df[j]==label)
                elif j==6:
                    x['?_logon type']=(df[j]==label)
            else:
                x[label]=(df[j]==label)
    for cl in ['source_class', 'destination_class']:
        for label in sorted(df[cl].unique()):
            if cl=='source_class':
                x['source_'+label]=(df[cl]==label)
            else:
                x['destination_'+label]=(df[cl]==label)
    X.append(x)
X[1]
X[0].columns
[len(entry.columns) for entry in X]
Explanation: I here repeat my procedure for generating labeled data and features for training/test data.
End of explanation
all_col = set(sum([list(entry.columns) for entry in X], []))
[all_col.difference(list(entry.columns)) for entry in X]
Explanation: I just discovered that my sample sets do not contain the same number of features. 
Below I am checking to see what the source of this difference.
End of explanation
col_set = [set(entry.columns) for entry in X]
common_subset = set.intersection(*col_set)
drop_cols = [e.difference(common_subset) for e in col_set]
for entry, to_drop in zip(X, drop_cols):
    print 'dropping', to_drop
    for item in to_drop:
        del entry[item]
col0 = list(X[0].columns)
for i in range(1,nfiles):
    col_i = list(X[i].columns)
    assert col0 == col_i, 'mismatch in %r:\n%s\n%s' % (i, col0, col_i)
Explanation: This is potentially different spelling of two different commands/labels. For now I will just remove all the labels that are not present in 15 files of data I have just downloaded. If the scores for machine learning will change noticeably. I will look into ways to clean and incorporate this data.
End of explanation
from sklearn import linear_model
clf_l1_LR = linear_model.LogisticRegression(C=1000, penalty='l1', tol=0.001).fit(X[0], Y[0])
scores=[]
scores.append(clf_l1_LR.score(X[0], Y[0]))
print 'score for training set', scores[0]
for i in range(1,nfiles):
    scores.append(clf_l1_LR.score(X[i], Y[i]))
    print 'score for test set', i, scores[i]
print 'mean', np.mean(scores), 'std', np.std(scores)
Explanation: Machine learning with logistic regression with Lasso
End of explanation
<END_TASK>