Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                    1                    0                    0                   0                          0                        0                        1                        0                        0                              0                       0                       0                       1                           0                           0                           0                           0                           0                           1                               0                     0                     0                     1                            0                         0                         1                          0                                0                       0                       1                       0                              0                           0                           1                           0                           0                                  0                                                 30                                  839.76                                 0.014384                              259.873954                               472.231119                                   86.49                              402.775486                                           0.000000                                7.928001                                 0.415426                           3.600926                                11.007471                                4.961414                                 1700.67                                                  5                                 0.602209                                 18                               51.149250                                94.481667                                   20.65                                 0.472342                                           0.000000                                 -0.333796                           -0.317685                                 0.204548                                  0.335175                                -0.470410                                  543.18                                                  5                                  128.51                                -0.539060                                  8                               36.734681                                66.666667                              1058.276667                                           5.000000                                139.960000                                  0.002397                           13.166667                                43.312326                                 78.705187                                14.415000                                          1                                            1                                         1                                        1                                                  5                                         1                                               0                                          1                                                 0                                          1                                              0                                      1                                             0                                                 1                                                 0                                                 0                                                  0                                               3                                              1                                            1                                             0                                              0                                            1                                            0                                            0                                            0                                            0                                                  0 
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                    0                    0                    0                   1                          0                        0                        0                        0                        1                              0                       1                       0                       0                           0                           1                           0                           0                           0                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       0                       0                       1                              0                           0                           0                           0                           1                                  0                                                 37                                 1157.99                                 0.002764                              356.125829                               649.657515                                  131.51                              235.992478                                           0.517549                                3.514421                                 0.387884                           3.335416                                13.027258                               16.960575                                 1351.46                                                  5                                 0.382868                                 18                               54.293903                               110.450000                                   54.83                                -0.391805                                          -0.644061                                  0.027256                            0.282488                                -1.065663                                  1.980948                                 2.103510                                  771.68                                                  4                                  139.20                                -0.711744                                 10                               29.026424                                70.638182                              1090.960000                                           4.625000                                144.748750                                  0.000346                           13.625000                                44.515729                                 81.207189                                16.438750                                          1                                            1                                         1                                        1                                                  5                                         1                                               0                                          1                                                 0                                          1                                              0                                      1                                             0                                                 0                                                 1                                                 0                                                  0                                               3                                              1                                            1                                             0                                              0                                            0                                            1                                            0                                            0                                            0                                                  0 
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                    1                    0                    0                   0                          0                        1                        0                        0                        0                              0                       1                       0                       0                           0                           0                           0                           1                           0                           0                               0                     1                     0                     0                            0                         0                         1                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                                 40                                 1057.97                                -0.476122                              312.745952                               582.193117                                   78.59                              279.510713                                           0.000000                                7.322191                                 0.589386                           4.062019                                13.759314                                6.954507                                 1613.93                                                  5                                 0.640252                                 25                               46.905665                                88.755625                                   26.36                                 0.778170                                           0.000000                                 -0.780493                            1.946018                                -0.312355                                 -0.424949                                 2.440005                                  809.97                                                  5                                  118.90                                -1.038434                                 12                               30.450261                                50.623125                              1128.202500                                           5.000000                                132.246250                                 -0.059515                           15.750000                                39.093244                                 72.774140                                 9.823750                                          1                                            1                                         1                                        1                                                  4                                         1                                               0                                          1                                                 0                                          1                                              0                                      1                                             0                                                 0                                                 0                                                 1                                                  0                                               3                                              1                                            1                                             0                                              0                                            0                                            0                                            0                                            0                                            1                                                  0 
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                    0                    0                    1                   0                          0                        0                        0                        1                        0                              0                       1                       0                       0                           0                           0                           1                           0                           0                           0                               0                     0                     1                     0                            0                         0                         0                          1                                0                       0                       1                       0                              0                           0                           0                           1                           0                                  0                                                 29                                  847.63                                 2.286086                              257.299895                               405.237462                                   66.21                              219.021420                                           0.408248                               10.724241                                 0.429374                           2.428992                                11.174282                                5.424407                                 1477.97                                                  5                                 0.854976                                 18                               50.110120                                82.109444                                   20.06                                 2.246479                                          -2.449490                                 -0.941078                           -1.507217                                -0.245703                                  0.678544                                 1.000771                                  889.21                                                  4                                  126.74                                -0.289466                                 11                               35.704680                                55.579412                              1039.436667                                           4.833333                                141.271667                                  0.381014                           15.500000                                42.883316                                 67.539577                                11.035000                                          1                                            1                                         1                                        1                                                  4                                         1                                               0                                          1                                                 0                                          1                                              0                                      1                                             0                                                 0                                                 1                                                 0                                                  0                                               3                                              1                                            0                                             1                                              0                                            0                                            0                                            1                                            0                                            0                                                  0 
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                    0                    1                    0                   0                          0                        1                        0                        0                        0                              0                       0                       1                       0                           0                           0                           0                           0                           1                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                                 35                                  931.63                                -0.277640                              258.700528                               548.905851                                  154.60                              251.609234                                           0.000000                               17.221593                                 0.509798                           3.450328                                11.477071                               15.874374                                 1320.64                                                  5                                 0.755711                                 18                               47.935920                                96.581000                                   56.46                                -0.440929                                           0.000000                                 -1.539467                           -0.303276                                 0.013087                                  0.235296                                 2.154929                                  634.84                                                  5                                  100.04                                -0.763603                                  8                               27.839228                                61.910000                              1028.611429                                           5.000000                                133.090000                                 -0.039663                           13.285714                                36.957218                                 78.415122                                22.085714                                          1                                            1                                         1                                        1                                                  4                                         1                                               0                                          1                                                 0                                          1                                              0                                      1                                             0                                                 1                                                 0                                                 0                                                  0                                               3                                              1                                            0                                             1                                              0                                            0                                            0                                            0                                            1                                            0                                                  0 

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 25                                  834.08                                -1.546156                              217.064024                               524.919674                                  198.92                              327.331994                                           1.329160                               11.742343                                 0.326628                           5.741661                                25.801702                               45.471977                                 1418.99                                                  5                                 0.164928                                 19                               51.192518                               136.672500                                  124.58                                 0.130163                                          -1.206607                                 -1.165955                           -0.243198                                -1.336611                                  1.796000                                 2.290693                                  546.69                                                  2                                  119.86                                -0.614442                                  4                                9.176046                                64.238824                               962.735000                                           4.166667                                139.013333                                 -0.257693                           12.166667                                36.177337                                 87.486612                                33.153333                                          1                                            1                                         1                                        1                                                  4                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                             False                                              True                                              False                                               3                                              1                                         True                                         False                                          False                                        False                                        False                                        False                                        False                                         True                                              False 
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                                 43                                 1180.90                                -1.797214                              319.497611                               733.862898                                  193.08                              333.923377                                           0.440959                               11.457114                                 0.324894                           4.272002                                11.875823                               17.851716                                 1650.65                                                  5                                 0.417250                                 21                               41.627134                               104.565000                                   60.29                                -0.385392                                          -1.619848                                  0.230847                           -0.086578                                -1.743267                                  1.078619                                 1.490781                                  557.32                                                  4                                  118.59                                -0.624344                                  8                               22.026552                                69.665000                              1130.936667                                           4.777778                                131.211111                                 -0.199690                           14.000000                                35.499735                                 81.540322                                21.453333                                          1                                            1                                         1                                        1                                                  5                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                         True                                        False                                        False                                        False                                              False 
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                                 23                                  715.80                                 0.353061                              215.417211                               407.390549                                  108.69                              450.318768                                           0.547723                                5.596950                                 0.530094                           4.868265                                10.012657                               10.829338                                 1691.27                                                  5                                 0.785376                                 19                               46.402508                                92.004000                                   38.69                                 0.727969                                          -0.608581                                 -1.598919                            0.570300                                -0.627238                                 -0.730554                                 1.010658                                  653.11                                                  4                                  133.75                                -0.608419                                  8                               37.945636                                66.563000                              1051.790000                                           4.600000                                143.160000                                  0.070612                           12.800000                                43.083442                                 81.478110                                21.738000                                          1                                            1                                         1                                        1                                                  2                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               2                                              1                                         True                                         False                                          False                                        False                                        False                                         True                                        False                                        False                                              False 
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 39                                 1100.82                                -0.082021                              315.817331                               615.714934                                  136.01                              346.152626                                           0.353553                                9.491736                                 0.305345                           4.086126                                10.757169                               10.517928                                 1690.97                                                  5                                 0.454842                                 21                               43.950396                                87.669412                                   40.88                                -0.576079                                          -2.828427                                 -0.964539                           -1.158217                                -0.403715                                 -2.048650                                 1.981423                                  619.93                                                  4                                  120.06                                -0.522578                                  8                               33.618728                                52.288421                              1234.708750                                           4.875000                                137.602500                                 -0.010253                           16.125000                                39.477166                                 76.964367                                17.001250                                          1                                            1                                         1                                        1                                                  4                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                        False                                        False                                         True                                        False                                              False 
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                                 33                                  997.48                                -0.595128                              261.535265                               584.302915                                  137.50                              263.424967                                           0.487950                                5.679896                                 0.440504                           2.699206                                 7.689676                               10.431664                                 1705.25                                                  5                                 0.538086                                 19                               44.978308                                95.488000                                   34.93                                 0.215180                                          -1.229634                                 -0.308063                           -0.566615                                -0.405840                                  0.246613                                 0.013052                                  921.73                                                  4                                  133.41                                -0.618136                                 11                               26.724118                                73.576471                              1288.391429                                           4.714286                                142.497143                                 -0.085018                           15.428571                                37.362181                                 83.471845                                19.642857                                          1                                            1                                         1                                        1                                                  4                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                         True                                        False                                        False                                        False                                        False                                              False 

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.device)  NUM_UNIQUE(transactions.sessions.customer_id)  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 25                                  834.08                                -1.546156                              217.064024                               524.919674                                  198.92                              327.331994                                           1.329160                               11.742343                                 0.326628                           5.741661                                25.801702                               45.471977                                 1418.99                                                  5                                 0.164928                                 19                               51.192518                               136.672500                                  124.58                                 0.130163                                          -1.206607                                 -1.165955                           -0.243198                                -1.336611                                  1.796000                                 2.290693                                  546.69                                                  2                                  119.86                                -0.614442                                  4                                9.176046                                64.238824                               962.735000                                           4.166667                                139.013333                                 -0.257693                           12.166667                                36.177337                                 87.486612                                33.153333                                          1                                            1                                         1                                        1                                                  4                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                             False                                              True                                              False                                               3                                              1                                         True                                         False                                          False                                        False                                        False                                        False                                        False                                         True                                              False 
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                                 43                                 1180.90                                -1.797214                              319.497611                               733.862898                                  193.08                              333.923377                                           0.440959                               11.457114                                 0.324894                           4.272002                                11.875823                               17.851716                                 1650.65                                                  5                                 0.417250                                 21                               41.627134                               104.565000                                   60.29                                -0.385392                                          -1.619848                                  0.230847                           -0.086578                                -1.743267                                  1.078619                                 1.490781                                  557.32                                                  4                                  118.59                                -0.624344                                  8                               22.026552                                69.665000                              1130.936667                                           4.777778                                131.211111                                 -0.199690                           14.000000                                35.499735                                 81.540322                                21.453333                                          1                                            1                                         1                                        1                                                  5                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                         True                                        False                                        False                                        False                                              False 
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                                 23                                  715.80                                 0.353061                              215.417211                               407.390549                                  108.69                              450.318768                                           0.547723                                5.596950                                 0.530094                           4.868265                                10.012657                               10.829338                                 1691.27                                                  5                                 0.785376                                 19                               46.402508                                92.004000                                   38.69                                 0.727969                                          -0.608581                                 -1.598919                            0.570300                                -0.627238                                 -0.730554                                 1.010658                                  653.11                                                  4                                  133.75                                -0.608419                                  8                               37.945636                                66.563000                              1051.790000                                           4.600000                                143.160000                                  0.070612                           12.800000                                43.083442                                 81.478110                                21.738000                                          1                                            1                                         1                                        1                                                  2                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               2                                              1                                         True                                         False                                          False                                        False                                        False                                         True                                        False                                        False                                              False 
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                                 39                                 1100.82                                -0.082021                              315.817331                               615.714934                                  136.01                              346.152626                                           0.353553                                9.491736                                 0.305345                           4.086126                                10.757169                               10.517928                                 1690.97                                                  5                                 0.454842                                 21                               43.950396                                87.669412                                   40.88                                -0.576079                                          -2.828427                                 -0.964539                           -1.158217                                -0.403715                                 -2.048650                                 1.981423                                  619.93                                                  4                                  120.06                                -0.522578                                  8                               33.618728                                52.288421                              1234.708750                                           4.875000                                137.602500                                 -0.010253                           16.125000                                39.477166                                 76.964367                                17.001250                                          1                                            1                                         1                                        1                                                  4                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                        False                                        False                                        False                                         True                                        False                                              False 
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                                 33                                  997.48                                -0.595128                              261.535265                               584.302915                                  137.50                              263.424967                                           0.487950                                5.679896                                 0.440504                           2.699206                                 7.689676                               10.431664                                 1705.25                                                  5                                 0.538086                                 19                               44.978308                                95.488000                                   34.93                                 0.215180                                          -1.229634                                 -0.308063                           -0.566615                                -0.405840                                  0.246613                                 0.013052                                  921.73                                                  4                                  133.41                                -0.618136                                 11                               26.724118                                73.576471                              1288.391429                                           4.714286                                142.497143                                 -0.085018                           15.428571                                37.362181                                 83.471845                                19.642857                                          1                                            1                                         1                                        1                                                  4                                      True                                           False                                       True                                             False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                               3                                              1                                        False                                          True                                          False                                         True                                        False                                        False                                        False                                        False                                              False