Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MIN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                    1                    0                    0                   0                          0                        0                        1                        0                        0                              0                       0                       0                       1                           0                           0                           0                           0                           0                           1                               0                     0                     0                     1                            0                         0                         1                          0                                0                       0                       1                       0                              0                           0                           1                           0                           0                                  0                                                 30                                   86.49                              259.873954                                  839.76                               472.231119                                 0.014384                           3.600926                              402.775486                                           0.000000                                4.961414                                7.928001                                11.007471                                 0.415426                                 18                                 1700.67                                                  5                                   20.65                               51.149250                                94.481667                                 0.602209                           -0.317685                                 0.472342                                           0.000000                                 -0.470410                                 0.204548                                -0.333796                                  0.335175                                  8                                  543.18                                                  5                               36.734681                                  128.51                                66.666667                                -0.539060                           13.166667                              1058.276667                                           5.000000                                 14.415000                                43.312326                               139.960000                                 78.705187                                  0.002397                                        1                                          1                                         1                                            1                                                  5                                       1                                             0                                        1                                               0                                          1                                              0                                          1                                                 0                                                 1                                                 0                                                 0                                                  0                                               3                                              1                                            1                                             0                                              0                                            1                                            0                                            0                                            0                                            0                                                  0 
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                    0                    0                    0                   1                          0                        0                        0                        0                        1                              0                       1                       0                       0                           0                           1                           0                           0                           0                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       0                       0                       1                              0                           0                           0                           0                           1                                  0                                                 37                                  131.51                              356.125829                                 1157.99                               649.657515                                 0.002764                           3.335416                              235.992478                                           0.517549                               16.960575                                3.514421                                13.027258                                 0.387884                                 18                                 1351.46                                                  5                                   54.83                               54.293903                               110.450000                                 0.382868                            0.282488                                -0.391805                                          -0.644061                                  2.103510                                -1.065663                                 0.027256                                  1.980948                                 10                                  771.68                                                  4                               29.026424                                  139.20                                70.638182                                -0.711744                           13.625000                              1090.960000                                           4.625000                                 16.438750                                44.515729                               144.748750                                 81.207189                                  0.000346                                        1                                          1                                         1                                            1                                                  5                                       1                                             0                                        1                                               0                                          1                                              0                                          1                                                 0                                                 0                                                 1                                                 0                                                  0                                               3                                              1                                            1                                             0                                              0                                            0                                            1                                            0                                            0                                            0                                                  0 
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                    1                    0                    0                   0                          0                        1                        0                        0                        0                              0                       1                       0                       0                           0                           0                           0                           1                           0                           0                               0                     1                     0                     0                            0                         0                         1                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                                 40                                   78.59                              312.745952                                 1057.97                               582.193117                                -0.476122                           4.062019                              279.510713                                           0.000000                                6.954507                                7.322191                                13.759314                                 0.589386                                 25                                 1613.93                                                  5                                   26.36                               46.905665                                88.755625                                 0.640252                            1.946018                                 0.778170                                           0.000000                                  2.440005                                -0.312355                                -0.780493                                 -0.424949                                 12                                  809.97                                                  5                               30.450261                                  118.90                                50.623125                                -1.038434                           15.750000                              1128.202500                                           5.000000                                  9.823750                                39.093244                               132.246250                                 72.774140                                 -0.059515                                        1                                          1                                         1                                            1                                                  4                                       1                                             0                                        1                                               0                                          1                                              0                                          1                                                 0                                                 0                                                 0                                                 1                                                  0                                               3                                              1                                            1                                             0                                              0                                            0                                            0                                            0                                            0                                            1                                                  0 
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                    0                    0                    1                   0                          0                        0                        0                        1                        0                              0                       1                       0                       0                           0                           0                           1                           0                           0                           0                               0                     0                     1                     0                            0                         0                         0                          1                                0                       0                       1                       0                              0                           0                           0                           1                           0                                  0                                                 29                                   66.21                              257.299895                                  847.63                               405.237462                                 2.286086                           2.428992                              219.021420                                           0.408248                                5.424407                               10.724241                                11.174282                                 0.429374                                 18                                 1477.97                                                  5                                   20.06                               50.110120                                82.109444                                 0.854976                           -1.507217                                 2.246479                                          -2.449490                                  1.000771                                -0.245703                                -0.941078                                  0.678544                                 11                                  889.21                                                  4                               35.704680                                  126.74                                55.579412                                -0.289466                           15.500000                              1039.436667                                           4.833333                                 11.035000                                42.883316                               141.271667                                 67.539577                                  0.381014                                        1                                          1                                         1                                            1                                                  4                                       1                                             0                                        1                                               0                                          1                                              0                                          1                                                 0                                                 0                                                 1                                                 0                                                  0                                               3                                              1                                            0                                             1                                              0                                            0                                            0                                            1                                            0                                            0                                                  0 
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                    0                    1                    0                   0                          0                        1                        0                        0                        0                              0                       0                       1                       0                           0                           0                           0                           0                           1                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                                 35                                  154.60                              258.700528                                  931.63                               548.905851                                -0.277640                           3.450328                              251.609234                                           0.000000                               15.874374                               17.221593                                11.477071                                 0.509798                                 18                                 1320.64                                                  5                                   56.46                               47.935920                                96.581000                                 0.755711                           -0.303276                                -0.440929                                           0.000000                                  2.154929                                 0.013087                                -1.539467                                  0.235296                                  8                                  634.84                                                  5                               27.839228                                  100.04                                61.910000                                -0.763603                           13.285714                              1028.611429                                           5.000000                                 22.085714                                36.957218                               133.090000                                 78.415122                                 -0.039663                                        1                                          1                                         1                                            1                                                  4                                       1                                             0                                        1                                               0                                          1                                              0                                          1                                                 0                                                 1                                                 0                                                 0                                                  0                                               3                                              1                                            0                                             1                                              0                                            0                                            0                                            0                                            1                                            0                                                  0 

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MIN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 25                                  198.92                              217.064024                                  834.08                               524.919674                                -1.546156                           5.741661                              327.331994                                           1.329160                               45.471977                               11.742343                                25.801702                                 0.326628                                 19                                 1418.99                                                  5                                  124.58                               51.192518                               136.672500                                 0.164928                           -0.243198                                 0.130163                                          -1.206607                                  2.290693                                -1.336611                                -1.165955                                  1.796000                                  4                                  546.69                                                  2                                9.176046                                  119.86                                64.238824                                -0.614442                           12.166667                               962.735000                                           4.166667                                 33.153333                                36.177337                               139.013333                                 87.486612                                 -0.257693                                        1                                          1                                         1                                            1                                                  4                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                             False                                              True                                              False                                               3                                              1                                         True                                         False                                          False                                        False                                        False                                        False                                        False                                         True                                              False 
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                                 43                                  193.08                              319.497611                                 1180.90                               733.862898                                -1.797214                           4.272002                              333.923377                                           0.440959                               17.851716                               11.457114                                11.875823                                 0.324894                                 21                                 1650.65                                                  5                                   60.29                               41.627134                               104.565000                                 0.417250                           -0.086578                                -0.385392                                          -1.619848                                  1.490781                                -1.743267                                 0.230847                                  1.078619                                  8                                  557.32                                                  4                               22.026552                                  118.59                                69.665000                                -0.624344                           14.000000                              1130.936667                                           4.777778                                 21.453333                                35.499735                               131.211111                                 81.540322                                 -0.199690                                        1                                          1                                         1                                            1                                                  5                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                         True                                        False                                        False                                        False                                              False 
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                                 23                                  108.69                              215.417211                                  715.80                               407.390549                                 0.353061                           4.868265                              450.318768                                           0.547723                               10.829338                                5.596950                                10.012657                                 0.530094                                 19                                 1691.27                                                  5                                   38.69                               46.402508                                92.004000                                 0.785376                            0.570300                                 0.727969                                          -0.608581                                  1.010658                                -0.627238                                -1.598919                                 -0.730554                                  8                                  653.11                                                  4                               37.945636                                  133.75                                66.563000                                -0.608419                           12.800000                              1051.790000                                           4.600000                                 21.738000                                43.083442                               143.160000                                 81.478110                                  0.070612                                        1                                          1                                         1                                            1                                                  2                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               2                                              1                                         True                                         False                                          False                                        False                                        False                                         True                                        False                                        False                                              False 
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 39                                  136.01                              315.817331                                 1100.82                               615.714934                                -0.082021                           4.086126                              346.152626                                           0.353553                               10.517928                                9.491736                                10.757169                                 0.305345                                 21                                 1690.97                                                  5                                   40.88                               43.950396                                87.669412                                 0.454842                           -1.158217                                -0.576079                                          -2.828427                                  1.981423                                -0.403715                                -0.964539                                 -2.048650                                  8                                  619.93                                                  4                               33.618728                                  120.06                                52.288421                                -0.522578                           16.125000                              1234.708750                                           4.875000                                 17.001250                                39.477166                               137.602500                                 76.964367                                 -0.010253                                        1                                          1                                         1                                            1                                                  4                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                        False                                        False                                         True                                        False                                              False 
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                                 33                                  137.50                              261.535265                                  997.48                               584.302915                                -0.595128                           2.699206                              263.424967                                           0.487950                               10.431664                                5.679896                                 7.689676                                 0.440504                                 19                                 1705.25                                                  5                                   34.93                               44.978308                                95.488000                                 0.538086                           -0.566615                                 0.215180                                          -1.229634                                  0.013052                                -0.405840                                -0.308063                                  0.246613                                 11                                  921.73                                                  4                               26.724118                                  133.41                                73.576471                                -0.618136                           15.428571                              1288.391429                                           4.714286                                 19.642857                                37.362181                               142.497143                                 83.471845                                 -0.085018                                        1                                          1                                         1                                            1                                                  4                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                         True                                        False                                        False                                        False                                        False                                              False 

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MIN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 25                                  198.92                              217.064024                                  834.08                               524.919674                                -1.546156                           5.741661                              327.331994                                           1.329160                               45.471977                               11.742343                                25.801702                                 0.326628                                 19                                 1418.99                                                  5                                  124.58                               51.192518                               136.672500                                 0.164928                           -0.243198                                 0.130163                                          -1.206607                                  2.290693                                -1.336611                                -1.165955                                  1.796000                                  4                                  546.69                                                  2                                9.176046                                  119.86                                64.238824                                -0.614442                           12.166667                               962.735000                                           4.166667                                 33.153333                                36.177337                               139.013333                                 87.486612                                 -0.257693                                        1                                          1                                         1                                            1                                                  4                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                             False                                              True                                              False                                               3                                              1                                         True                                         False                                          False                                        False                                        False                                        False                                        False                                         True                                              False 
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                                 43                                  193.08                              319.497611                                 1180.90                               733.862898                                -1.797214                           4.272002                              333.923377                                           0.440959                               17.851716                               11.457114                                11.875823                                 0.324894                                 21                                 1650.65                                                  5                                   60.29                               41.627134                               104.565000                                 0.417250                           -0.086578                                -0.385392                                          -1.619848                                  1.490781                                -1.743267                                 0.230847                                  1.078619                                  8                                  557.32                                                  4                               22.026552                                  118.59                                69.665000                                -0.624344                           14.000000                              1130.936667                                           4.777778                                 21.453333                                35.499735                               131.211111                                 81.540322                                 -0.199690                                        1                                          1                                         1                                            1                                                  5                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                         True                                        False                                        False                                        False                                              False 
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                                 23                                  108.69                              215.417211                                  715.80                               407.390549                                 0.353061                           4.868265                              450.318768                                           0.547723                               10.829338                                5.596950                                10.012657                                 0.530094                                 19                                 1691.27                                                  5                                   38.69                               46.402508                                92.004000                                 0.785376                            0.570300                                 0.727969                                          -0.608581                                  1.010658                                -0.627238                                -1.598919                                 -0.730554                                  8                                  653.11                                                  4                               37.945636                                  133.75                                66.563000                                -0.608419                           12.800000                              1051.790000                                           4.600000                                 21.738000                                43.083442                               143.160000                                 81.478110                                  0.070612                                        1                                          1                                         1                                            1                                                  2                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               2                                              1                                         True                                         False                                          False                                        False                                        False                                         True                                        False                                        False                                              False 
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 39                                  136.01                              315.817331                                 1100.82                               615.714934                                -0.082021                           4.086126                              346.152626                                           0.353553                               10.517928                                9.491736                                10.757169                                 0.305345                                 21                                 1690.97                                                  5                                   40.88                               43.950396                                87.669412                                 0.454842                           -1.158217                                -0.576079                                          -2.828427                                  1.981423                                -0.403715                                -0.964539                                 -2.048650                                  8                                  619.93                                                  4                               33.618728                                  120.06                                52.288421                                -0.522578                           16.125000                              1234.708750                                           4.875000                                 17.001250                                39.477166                               137.602500                                 76.964367                                 -0.010253                                        1                                          1                                         1                                            1                                                  4                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                        False                                        False                                         True                                        False                                              False 
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                                 33                                  137.50                              261.535265                                  997.48                               584.302915                                -0.595128                           2.699206                              263.424967                                           0.487950                               10.431664                                5.679896                                 7.689676                                 0.440504                                 19                                 1705.25                                                  5                                   34.93                               44.978308                                95.488000                                 0.538086                           -0.566615                                 0.215180                                          -1.229634                                  0.013052                                -0.405840                                -0.308063                                  0.246613                                 11                                  921.73                                                  4                               26.724118                                  133.41                                73.576471                                -0.618136                           15.428571                              1288.391429                                           4.714286                                 19.642857                                37.362181                               142.497143                                 83.471845                                 -0.085018                                        1                                          1                                         1                                            1                                                  4                                    True                                         False                                     True                                           False                                       True                                          False                                       True                                             False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                         True                                        False                                        False                                        False                                        False                                              False