Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.STD(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MIN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MIN(transactions.amount))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                    1                    0                    0                   0                          0                        0                        1                        0                        0                              0                       0                       0                       1                           0                           0                           0                           0                           0                           1                               0                     0                     0                     1                            0                         0                         1                          0                                0                       0                       1                       0                              0                           0                           1                           0                           0                                  0                              259.873954                                                 30                               472.231119                                  839.76                                 0.014384                                   86.49                                           0.000000                                11.007471                                7.928001                                 0.415426                              402.775486                           3.600926                                4.961414                               51.149250                                                  5                                94.481667                                 0.602209                                 1700.67                                 18                                   20.65                                 0.204548                                           0.000000                                   0.335175                                -0.333796                                 0.472342                           -0.317685                                -0.470410                               36.734681                                                  5                                66.666667                                  128.51                                -0.539060                                  543.18                                  8                                43.312326                                           5.000000                                  78.705187                               139.960000                                  0.002397                              1058.276667                           13.166667                                14.415000                                                  5                                           1                                        1                                            1                                         1                                                 1                                                 0                                                 0                                                  0                                              1                                               0                                      1                                             0                                          1                                                 0                                          1                                              0                                         3                                              1                                            1                                             0                                              0                                            1                                            0                                            0                                            0                                            0                                                  0 
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                    0                    0                    0                   1                          0                        0                        0                        0                        1                              0                       1                       0                       0                           0                           1                           0                           0                           0                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       0                       0                       1                              0                           0                           0                           0                           1                                  0                              356.125829                                                 37                               649.657515                                 1157.99                                 0.002764                                  131.51                                           0.517549                                13.027258                                3.514421                                 0.387884                              235.992478                           3.335416                               16.960575                               54.293903                                                  5                               110.450000                                 0.382868                                 1351.46                                 18                                   54.83                                -1.065663                                          -0.644061                                   1.980948                                 0.027256                                -0.391805                            0.282488                                 2.103510                               29.026424                                                  4                                70.638182                                  139.20                                -0.711744                                  771.68                                 10                                44.515729                                           4.625000                                  81.207189                               144.748750                                  0.000346                              1090.960000                           13.625000                                16.438750                                                  5                                           1                                        1                                            1                                         1                                                 0                                                 1                                                 0                                                  0                                              1                                               0                                      1                                             0                                          1                                                 0                                          1                                              0                                         3                                              1                                            1                                             0                                              0                                            0                                            1                                            0                                            0                                            0                                                  0 
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                    1                    0                    0                   0                          0                        1                        0                        0                        0                              0                       1                       0                       0                           0                           0                           0                           1                           0                           0                               0                     1                     0                     0                            0                         0                         1                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                              312.745952                                                 40                               582.193117                                 1057.97                                -0.476122                                   78.59                                           0.000000                                13.759314                                7.322191                                 0.589386                              279.510713                           4.062019                                6.954507                               46.905665                                                  5                                88.755625                                 0.640252                                 1613.93                                 25                                   26.36                                -0.312355                                           0.000000                                  -0.424949                                -0.780493                                 0.778170                            1.946018                                 2.440005                               30.450261                                                  5                                50.623125                                  118.90                                -1.038434                                  809.97                                 12                                39.093244                                           5.000000                                  72.774140                               132.246250                                 -0.059515                              1128.202500                           15.750000                                 9.823750                                                  4                                           1                                        1                                            1                                         1                                                 0                                                 0                                                 1                                                  0                                              1                                               0                                      1                                             0                                          1                                                 0                                          1                                              0                                         3                                              1                                            1                                             0                                              0                                            0                                            0                                            0                                            0                                            1                                                  0 
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                    0                    0                    1                   0                          0                        0                        0                        1                        0                              0                       1                       0                       0                           0                           0                           1                           0                           0                           0                               0                     0                     1                     0                            0                         0                         0                          1                                0                       0                       1                       0                              0                           0                           0                           1                           0                                  0                              257.299895                                                 29                               405.237462                                  847.63                                 2.286086                                   66.21                                           0.408248                                11.174282                               10.724241                                 0.429374                              219.021420                           2.428992                                5.424407                               50.110120                                                  5                                82.109444                                 0.854976                                 1477.97                                 18                                   20.06                                -0.245703                                          -2.449490                                   0.678544                                -0.941078                                 2.246479                           -1.507217                                 1.000771                               35.704680                                                  4                                55.579412                                  126.74                                -0.289466                                  889.21                                 11                                42.883316                                           4.833333                                  67.539577                               141.271667                                  0.381014                              1039.436667                           15.500000                                11.035000                                                  4                                           1                                        1                                            1                                         1                                                 0                                                 1                                                 0                                                  0                                              1                                               0                                      1                                             0                                          1                                                 0                                          1                                              0                                         3                                              1                                            0                                             1                                              0                                            0                                            0                                            1                                            0                                            0                                                  0 
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                    0                    1                    0                   0                          0                        1                        0                        0                        0                              0                       0                       1                       0                           0                           0                           0                           0                           1                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                              258.700528                                                 35                               548.905851                                  931.63                                -0.277640                                  154.60                                           0.000000                                11.477071                               17.221593                                 0.509798                              251.609234                           3.450328                               15.874374                               47.935920                                                  5                                96.581000                                 0.755711                                 1320.64                                 18                                   56.46                                 0.013087                                           0.000000                                   0.235296                                -1.539467                                -0.440929                           -0.303276                                 2.154929                               27.839228                                                  5                                61.910000                                  100.04                                -0.763603                                  634.84                                  8                                36.957218                                           5.000000                                  78.415122                               133.090000                                 -0.039663                              1028.611429                           13.285714                                22.085714                                                  4                                           1                                        1                                            1                                         1                                                 1                                                 0                                                 0                                                  0                                              1                                               0                                      1                                             0                                          1                                                 0                                          1                                              0                                         3                                              1                                            0                                             1                                              0                                            0                                            0                                            0                                            1                                            0                                                  0 

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.STD(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MIN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MIN(transactions.amount))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                              217.064024                                                 25                               524.919674                                  834.08                                -1.546156                                  198.92                                           1.329160                                25.801702                               11.742343                                 0.326628                              327.331994                           5.741661                               45.471977                               51.192518                                                  5                               136.672500                                 0.164928                                 1418.99                                 19                                  124.58                                -1.336611                                          -1.206607                                   1.796000                                -1.165955                                 0.130163                           -0.243198                                 2.290693                                9.176046                                                  2                                64.238824                                  119.86                                -0.614442                                  546.69                                  4                                36.177337                                           4.166667                                  87.486612                               139.013333                                 -0.257693                               962.735000                           12.166667                                33.153333                                                  4                                           1                                        1                                            1                                         1                                             False                                             False                                              True                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                         True                                         False                                          False                                        False                                        False                                        False                                        False                                         True                                              False 
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                              319.497611                                                 43                               733.862898                                 1180.90                                -1.797214                                  193.08                                           0.440959                                11.875823                               11.457114                                 0.324894                              333.923377                           4.272002                               17.851716                               41.627134                                                  5                               104.565000                                 0.417250                                 1650.65                                 21                                   60.29                                -1.743267                                          -1.619848                                   1.078619                                 0.230847                                -0.385392                           -0.086578                                 1.490781                               22.026552                                                  4                                69.665000                                  118.59                                -0.624344                                  557.32                                  8                                35.499735                                           4.777778                                  81.540322                               131.211111                                 -0.199690                              1130.936667                           14.000000                                21.453333                                                  5                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                        False                                          True                                          False                                        False                                         True                                        False                                        False                                        False                                              False 
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                              215.417211                                                 23                               407.390549                                  715.80                                 0.353061                                  108.69                                           0.547723                                10.012657                                5.596950                                 0.530094                              450.318768                           4.868265                               10.829338                               46.402508                                                  5                                92.004000                                 0.785376                                 1691.27                                 19                                   38.69                                -0.627238                                          -0.608581                                  -0.730554                                -1.598919                                 0.727969                            0.570300                                 1.010658                               37.945636                                                  4                                66.563000                                  133.75                                -0.608419                                  653.11                                  8                                43.083442                                           4.600000                                  81.478110                               143.160000                                  0.070612                              1051.790000                           12.800000                                21.738000                                                  2                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         2                                              1                                         True                                         False                                          False                                        False                                        False                                         True                                        False                                        False                                              False 
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                              315.817331                                                 39                               615.714934                                 1100.82                                -0.082021                                  136.01                                           0.353553                                10.757169                                9.491736                                 0.305345                              346.152626                           4.086126                               10.517928                               43.950396                                                  5                                87.669412                                 0.454842                                 1690.97                                 21                                   40.88                                -0.403715                                          -2.828427                                  -2.048650                                -0.964539                                -0.576079                           -1.158217                                 1.981423                               33.618728                                                  4                                52.288421                                  120.06                                -0.522578                                  619.93                                  8                                39.477166                                           4.875000                                  76.964367                               137.602500                                 -0.010253                              1234.708750                           16.125000                                17.001250                                                  4                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                        False                                          True                                          False                                        False                                        False                                        False                                         True                                        False                                              False 
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                              261.535265                                                 33                               584.302915                                  997.48                                -0.595128                                  137.50                                           0.487950                                 7.689676                                5.679896                                 0.440504                              263.424967                           2.699206                               10.431664                               44.978308                                                  5                                95.488000                                 0.538086                                 1705.25                                 19                                   34.93                                -0.405840                                          -1.229634                                   0.246613                                -0.308063                                 0.215180                           -0.566615                                 0.013052                               26.724118                                                  4                                73.576471                                  133.41                                -0.618136                                  921.73                                 11                                37.362181                                           4.714286                                  83.471845                               142.497143                                 -0.085018                              1288.391429                           15.428571                                19.642857                                                  4                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                        False                                          True                                          False                                         True                                        False                                        False                                        False                                        False                                              False 

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.STD(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MIN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MIN(transactions.amount))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                              217.064024                                                 25                               524.919674                                  834.08                                -1.546156                                  198.92                                           1.329160                                25.801702                               11.742343                                 0.326628                              327.331994                           5.741661                               45.471977                               51.192518                                                  5                               136.672500                                 0.164928                                 1418.99                                 19                                  124.58                                -1.336611                                          -1.206607                                   1.796000                                -1.165955                                 0.130163                           -0.243198                                 2.290693                                9.176046                                                  2                                64.238824                                  119.86                                -0.614442                                  546.69                                  4                                36.177337                                           4.166667                                  87.486612                               139.013333                                 -0.257693                               962.735000                           12.166667                                33.153333                                                  4                                           1                                        1                                            1                                         1                                             False                                             False                                              True                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                         True                                         False                                          False                                        False                                        False                                        False                                        False                                         True                                              False 
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                              319.497611                                                 43                               733.862898                                 1180.90                                -1.797214                                  193.08                                           0.440959                                11.875823                               11.457114                                 0.324894                              333.923377                           4.272002                               17.851716                               41.627134                                                  5                               104.565000                                 0.417250                                 1650.65                                 21                                   60.29                                -1.743267                                          -1.619848                                   1.078619                                 0.230847                                -0.385392                           -0.086578                                 1.490781                               22.026552                                                  4                                69.665000                                  118.59                                -0.624344                                  557.32                                  8                                35.499735                                           4.777778                                  81.540322                               131.211111                                 -0.199690                              1130.936667                           14.000000                                21.453333                                                  5                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                        False                                          True                                          False                                        False                                         True                                        False                                        False                                        False                                              False 
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                              215.417211                                                 23                               407.390549                                  715.80                                 0.353061                                  108.69                                           0.547723                                10.012657                                5.596950                                 0.530094                              450.318768                           4.868265                               10.829338                               46.402508                                                  5                                92.004000                                 0.785376                                 1691.27                                 19                                   38.69                                -0.627238                                          -0.608581                                  -0.730554                                -1.598919                                 0.727969                            0.570300                                 1.010658                               37.945636                                                  4                                66.563000                                  133.75                                -0.608419                                  653.11                                  8                                43.083442                                           4.600000                                  81.478110                               143.160000                                  0.070612                              1051.790000                           12.800000                                21.738000                                                  2                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         2                                              1                                         True                                         False                                          False                                        False                                        False                                         True                                        False                                        False                                              False 
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                              315.817331                                                 39                               615.714934                                 1100.82                                -0.082021                                  136.01                                           0.353553                                10.757169                                9.491736                                 0.305345                              346.152626                           4.086126                               10.517928                               43.950396                                                  5                                87.669412                                 0.454842                                 1690.97                                 21                                   40.88                                -0.403715                                          -2.828427                                  -2.048650                                -0.964539                                -0.576079                           -1.158217                                 1.981423                               33.618728                                                  4                                52.288421                                  120.06                                -0.522578                                  619.93                                  8                                39.477166                                           4.875000                                  76.964367                               137.602500                                 -0.010253                              1234.708750                           16.125000                                17.001250                                                  4                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                        False                                          True                                          False                                        False                                        False                                        False                                         True                                        False                                              False 
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                              261.535265                                                 33                               584.302915                                  997.48                                -0.595128                                  137.50                                           0.487950                                 7.689676                                5.679896                                 0.440504                              263.424967                           2.699206                               10.431664                               44.978308                                                  5                                95.488000                                 0.538086                                 1705.25                                 19                                   34.93                                -0.405840                                          -1.229634                                   0.246613                                -0.308063                                 0.215180                           -0.566615                                 0.013052                               26.724118                                                  4                                73.576471                                  133.41                                -0.618136                                  921.73                                 11                                37.362181                                           4.714286                                  83.471845                               142.497143                                 -0.085018                              1288.391429                           15.428571                                19.642857                                                  4                                           1                                        1                                            1                                         1                                             False                                              True                                             False                                              False                                           True                                           False                                   True                                         False                                       True                                             False                                       True                                          False                                         3                                              1                                        False                                          True                                          False                                         True                                        False                                        False                                        False                                        False                                              False