Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                    1                    0                    0                   0                          0                        0                        1                        0                        0                              0                       0                       0                       1                           0                           0                           0                           0                           0                           1                               0                     0                     0                     1                            0                         0                         1                          0                                0                       0                       1                       0                              0                           0                           1                           0                           0                                  0                                 0.014384                               472.231119                                  839.76                                   86.49                              259.873954                                                 30                                 0.415426                                11.007471                              402.775486                           3.600926                                7.928001                                4.961414                                           0.000000                                 0.602209                                94.481667                                 1700.67                                 18                                   20.65                               51.149250                                                  5                                  0.335175                                 0.472342                           -0.317685                                -0.333796                                -0.470410                                 0.204548                                           0.000000                                 -0.539060                                66.666667                                  543.18                                  8                                  128.51                               36.734681                                                  5                                  0.002397                                 78.705187                              1058.276667                           13.166667                               139.960000                                14.415000                                43.312326                                           5.000000                                           1                                         1                                                  5                                         1                                            1                                        1                                               0                                          1                                              0                                                 1                                                 0                                                 0                                                  0                                            1                                             0                                          1                                                 0                                              1                                         3                                            1                                            0                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                    0                    0                    0                   1                          0                        0                        0                        0                        1                              0                       1                       0                       0                           0                           1                           0                           0                           0                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       0                       0                       1                              0                           0                           0                           0                           1                                  0                                 0.002764                               649.657515                                 1157.99                                  131.51                              356.125829                                                 37                                 0.387884                                13.027258                              235.992478                           3.335416                                3.514421                               16.960575                                           0.517549                                 0.382868                               110.450000                                 1351.46                                 18                                   54.83                               54.293903                                                  5                                  1.980948                                -0.391805                            0.282488                                 0.027256                                 2.103510                                -1.065663                                          -0.644061                                 -0.711744                                70.638182                                  771.68                                 10                                  139.20                               29.026424                                                  4                                  0.000346                                 81.207189                              1090.960000                           13.625000                               144.748750                                16.438750                                44.515729                                           4.625000                                           1                                         1                                                  5                                         1                                            1                                        1                                               0                                          1                                              0                                                 0                                                 1                                                 0                                                  0                                            1                                             0                                          1                                                 0                                              1                                         3                                            0                                            1                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                    1                    0                    0                   0                          0                        1                        0                        0                        0                              0                       1                       0                       0                           0                           0                           0                           1                           0                           0                               0                     1                     0                     0                            0                         0                         1                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                -0.476122                               582.193117                                 1057.97                                   78.59                              312.745952                                                 40                                 0.589386                                13.759314                              279.510713                           4.062019                                7.322191                                6.954507                                           0.000000                                 0.640252                                88.755625                                 1613.93                                 25                                   26.36                               46.905665                                                  5                                 -0.424949                                 0.778170                            1.946018                                -0.780493                                 2.440005                                -0.312355                                           0.000000                                 -1.038434                                50.623125                                  809.97                                 12                                  118.90                               30.450261                                                  5                                 -0.059515                                 72.774140                              1128.202500                           15.750000                               132.246250                                 9.823750                                39.093244                                           5.000000                                           1                                         1                                                  4                                         1                                            1                                        1                                               0                                          1                                              0                                                 0                                                 0                                                 1                                                  0                                            1                                             0                                          1                                                 0                                              1                                         3                                            0                                            0                                            0                                            0                                            1                                                  0                                             1                                             0                                              0
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                    0                    0                    1                   0                          0                        0                        0                        1                        0                              0                       1                       0                       0                           0                           0                           1                           0                           0                           0                               0                     0                     1                     0                            0                         0                         0                          1                                0                       0                       1                       0                              0                           0                           0                           1                           0                                  0                                 2.286086                               405.237462                                  847.63                                   66.21                              257.299895                                                 29                                 0.429374                                11.174282                              219.021420                           2.428992                               10.724241                                5.424407                                           0.408248                                 0.854976                                82.109444                                 1477.97                                 18                                   20.06                               50.110120                                                  5                                  0.678544                                 2.246479                           -1.507217                                -0.941078                                 1.000771                                -0.245703                                          -2.449490                                 -0.289466                                55.579412                                  889.21                                 11                                  126.74                               35.704680                                                  4                                  0.381014                                 67.539577                              1039.436667                           15.500000                               141.271667                                11.035000                                42.883316                                           4.833333                                           1                                         1                                                  4                                         1                                            1                                        1                                               0                                          1                                              0                                                 0                                                 1                                                 0                                                  0                                            1                                             0                                          1                                                 0                                              1                                         3                                            0                                            0                                            1                                            0                                            0                                                  0                                             0                                             1                                              0
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                    0                    1                    0                   0                          0                        1                        0                        0                        0                              0                       0                       1                       0                           0                           0                           0                           0                           1                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                -0.277640                               548.905851                                  931.63                                  154.60                              258.700528                                                 35                                 0.509798                                11.477071                              251.609234                           3.450328                               17.221593                               15.874374                                           0.000000                                 0.755711                                96.581000                                 1320.64                                 18                                   56.46                               47.935920                                                  5                                  0.235296                                -0.440929                           -0.303276                                -1.539467                                 2.154929                                 0.013087                                           0.000000                                 -0.763603                                61.910000                                  634.84                                  8                                  100.04                               27.839228                                                  5                                 -0.039663                                 78.415122                              1028.611429                           13.285714                               133.090000                                22.085714                                36.957218                                           5.000000                                           1                                         1                                                  4                                         1                                            1                                        1                                               0                                          1                                              0                                                 1                                                 0                                                 0                                                  0                                            1                                             0                                          1                                                 0                                              1                                         3                                            0                                            0                                            0                                            1                                            0                                                  0                                             0                                             1                                              0

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                -1.546156                               524.919674                                  834.08                                  198.92                              217.064024                                                 25                                 0.326628                                25.801702                              327.331994                           5.741661                               11.742343                               45.471977                                           1.329160                                 0.164928                               136.672500                                 1418.99                                 19                                  124.58                               51.192518                                                  5                                  1.796000                                 0.130163                           -0.243198                                -1.165955                                 2.290693                                -1.336611                                          -1.206607                                 -0.614442                                64.238824                                  546.69                                  4                                  119.86                                9.176046                                                  2                                 -0.257693                                 87.486612                               962.735000                           12.166667                               139.013333                                33.153333                                36.177337                                           4.166667                                           1                                         1                                                  4                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                             False                                              True                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                -1.797214                               733.862898                                 1180.90                                  193.08                              319.497611                                                 43                                 0.324894                                11.875823                              333.923377                           4.272002                               11.457114                               17.851716                                           0.440959                                 0.417250                               104.565000                                 1650.65                                 21                                   60.29                               41.627134                                                  5                                  1.078619                                -0.385392                           -0.086578                                 0.230847                                 1.490781                                -1.743267                                          -1.619848                                 -0.624344                                69.665000                                  557.32                                  8                                  118.59                               22.026552                                                  4                                 -0.199690                                 81.540322                              1130.936667                           14.000000                               131.211111                                21.453333                                35.499735                                           4.777778                                           1                                         1                                                  5                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                 0.353061                               407.390549                                  715.80                                  108.69                              215.417211                                                 23                                 0.530094                                10.012657                              450.318768                           4.868265                                5.596950                               10.829338                                           0.547723                                 0.785376                                92.004000                                 1691.27                                 19                                   38.69                               46.402508                                                  5                                 -0.730554                                 0.727969                            0.570300                                -1.598919                                 1.010658                                -0.627238                                          -0.608581                                 -0.608419                                66.563000                                  653.11                                  8                                  133.75                               37.945636                                                  4                                  0.070612                                 81.478110                              1051.790000                           12.800000                               143.160000                                21.738000                                43.083442                                           4.600000                                           1                                         1                                                  2                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                -0.082021                               615.714934                                 1100.82                                  136.01                              315.817331                                                 39                                 0.305345                                10.757169                              346.152626                           4.086126                                9.491736                               10.517928                                           0.353553                                 0.454842                                87.669412                                 1690.97                                 21                                   40.88                               43.950396                                                  5                                 -2.048650                                -0.576079                           -1.158217                                -0.964539                                 1.981423                                -0.403715                                          -2.828427                                 -0.522578                                52.288421                                  619.93                                  8                                  120.06                               33.618728                                                  4                                 -0.010253                                 76.964367                              1234.708750                           16.125000                               137.602500                                17.001250                                39.477166                                           4.875000                                           1                                         1                                                  4                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                -0.595128                               584.302915                                  997.48                                  137.50                              261.535265                                                 33                                 0.440504                                 7.689676                              263.424967                           2.699206                                5.679896                               10.431664                                           0.487950                                 0.538086                                95.488000                                 1705.25                                 19                                   34.93                               44.978308                                                  5                                  0.246613                                 0.215180                           -0.566615                                -0.308063                                 0.013052                                -0.405840                                          -1.229634                                 -0.618136                                73.576471                                  921.73                                 11                                  133.41                               26.724118                                                  4                                 -0.085018                                 83.471845                              1288.391429                           15.428571                               142.497143                                19.642857                                37.362181                                           4.714286                                           1                                         1                                                  4                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                -1.546156                               524.919674                                  834.08                                  198.92                              217.064024                                                 25                                 0.326628                                25.801702                              327.331994                           5.741661                               11.742343                               45.471977                                           1.329160                                 0.164928                               136.672500                                 1418.99                                 19                                  124.58                               51.192518                                                  5                                  1.796000                                 0.130163                           -0.243198                                -1.165955                                 2.290693                                -1.336611                                          -1.206607                                 -0.614442                                64.238824                                  546.69                                  4                                  119.86                                9.176046                                                  2                                 -0.257693                                 87.486612                               962.735000                           12.166667                               139.013333                                33.153333                                36.177337                                           4.166667                                           1                                         1                                                  4                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                             False                                              True                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                -1.797214                               733.862898                                 1180.90                                  193.08                              319.497611                                                 43                                 0.324894                                11.875823                              333.923377                           4.272002                               11.457114                               17.851716                                           0.440959                                 0.417250                               104.565000                                 1650.65                                 21                                   60.29                               41.627134                                                  5                                  1.078619                                -0.385392                           -0.086578                                 0.230847                                 1.490781                                -1.743267                                          -1.619848                                 -0.624344                                69.665000                                  557.32                                  8                                  118.59                               22.026552                                                  4                                 -0.199690                                 81.540322                              1130.936667                           14.000000                               131.211111                                21.453333                                35.499735                                           4.777778                                           1                                         1                                                  5                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                 0.353061                               407.390549                                  715.80                                  108.69                              215.417211                                                 23                                 0.530094                                10.012657                              450.318768                           4.868265                                5.596950                               10.829338                                           0.547723                                 0.785376                                92.004000                                 1691.27                                 19                                   38.69                               46.402508                                                  5                                 -0.730554                                 0.727969                            0.570300                                -1.598919                                 1.010658                                -0.627238                                          -0.608581                                 -0.608419                                66.563000                                  653.11                                  8                                  133.75                               37.945636                                                  4                                  0.070612                                 81.478110                              1051.790000                           12.800000                               143.160000                                21.738000                                43.083442                                           4.600000                                           1                                         1                                                  2                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                -0.082021                               615.714934                                 1100.82                                  136.01                              315.817331                                                 39                                 0.305345                                10.757169                              346.152626                           4.086126                                9.491736                               10.517928                                           0.353553                                 0.454842                                87.669412                                 1690.97                                 21                                   40.88                               43.950396                                                  5                                 -2.048650                                -0.576079                           -1.158217                                -0.964539                                 1.981423                                -0.403715                                          -2.828427                                 -0.522578                                52.288421                                  619.93                                  8                                  120.06                               33.618728                                                  4                                 -0.010253                                 76.964367                              1234.708750                           16.125000                               137.602500                                17.001250                                39.477166                                           4.875000                                           1                                         1                                                  4                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                -0.595128                               584.302915                                  997.48                                  137.50                              261.535265                                                 33                                 0.440504                                 7.689676                              263.424967                           2.699206                                5.679896                               10.431664                                           0.487950                                 0.538086                                95.488000                                 1705.25                                 19                                   34.93                               44.978308                                                  5                                  0.246613                                 0.215180                           -0.566615                                -0.308063                                 0.013052                                -0.405840                                          -1.229634                                 -0.618136                                73.576471                                  921.73                                 11                                  133.41                               26.724118                                                  4                                 -0.085018                                 83.471845                              1288.391429                           15.428571                               142.497143                                19.642857                                37.362181                                           4.714286                                           1                                         1                                                  4                                         1                                            1                                     True                                           False                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                       True                                             False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False