Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SUM(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.COUNT(transactions))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.COUNT(transactions))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.COUNT(transactions))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.COUNT(transactions))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.COUNT(transactions))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                    1                    0                    0                   0                          0                        0                        1                        0                        0                              0                       0                       0                       1                           0                           0                           0                           0                           0                           1                               0                     0                     0                     1                            0                         0                         1                          0                                0                       0                       1                       0                              0                           0                           1                           0                           0                                  0                               472.231119                              259.873954                                   86.49                                 0.014384                                  839.76                                                 30                              402.775486                                11.007471                                4.961414                                 0.415426                                7.928001                                           0.000000                           3.600926                                 1700.67                                94.481667                               51.149250                                   20.65                                 0.602209                                                  5                                 18                                 0.472342                                  0.335175                                 0.204548                                -0.470410                                -0.333796                                           0.000000                            -0.317685                                  543.18                                66.666667                               36.734681                                -0.539060                                  128.51                                                  5                                  8                              1058.276667                                 78.705187                                43.312326                                14.415000                                  0.002397                               139.960000                                           5.000000                            13.166667                                         1                                                  5                                         1                                          1                                            1                                          1                                              0                                                 1                                                 0                                                 0                                                  0                                            1                                             0                                        1                                               0                                          1                                                 0                                              1                                         3                                            1                                            0                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                    0                    0                    0                   1                          0                        0                        0                        0                        1                              0                       1                       0                       0                           0                           1                           0                           0                           0                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       0                       0                       1                              0                           0                           0                           0                           1                                  0                               649.657515                              356.125829                                  131.51                                 0.002764                                 1157.99                                                 37                              235.992478                                13.027258                               16.960575                                 0.387884                                3.514421                                           0.517549                           3.335416                                 1351.46                               110.450000                               54.293903                                   54.83                                 0.382868                                                  5                                 18                                -0.391805                                  1.980948                                -1.065663                                 2.103510                                 0.027256                                          -0.644061                             0.282488                                  771.68                                70.638182                               29.026424                                -0.711744                                  139.20                                                  4                                 10                              1090.960000                                 81.207189                                44.515729                                16.438750                                  0.000346                               144.748750                                           4.625000                            13.625000                                         1                                                  5                                         1                                          1                                            1                                          1                                              0                                                 0                                                 1                                                 0                                                  0                                            1                                             0                                        1                                               0                                          1                                                 0                                              1                                         3                                            0                                            1                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                    1                    0                    0                   0                          0                        1                        0                        0                        0                              0                       1                       0                       0                           0                           0                           0                           1                           0                           0                               0                     1                     0                     0                            0                         0                         1                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                               582.193117                              312.745952                                   78.59                                -0.476122                                 1057.97                                                 40                              279.510713                                13.759314                                6.954507                                 0.589386                                7.322191                                           0.000000                           4.062019                                 1613.93                                88.755625                               46.905665                                   26.36                                 0.640252                                                  5                                 25                                 0.778170                                 -0.424949                                -0.312355                                 2.440005                                -0.780493                                           0.000000                             1.946018                                  809.97                                50.623125                               30.450261                                -1.038434                                  118.90                                                  5                                 12                              1128.202500                                 72.774140                                39.093244                                 9.823750                                 -0.059515                               132.246250                                           5.000000                            15.750000                                         1                                                  4                                         1                                          1                                            1                                          1                                              0                                                 0                                                 0                                                 1                                                  0                                            1                                             0                                        1                                               0                                          1                                                 0                                              1                                         3                                            0                                            0                                            0                                            0                                            1                                                  0                                             1                                             0                                              0
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                    0                    0                    1                   0                          0                        0                        0                        1                        0                              0                       1                       0                       0                           0                           0                           1                           0                           0                           0                               0                     0                     1                     0                            0                         0                         0                          1                                0                       0                       1                       0                              0                           0                           0                           1                           0                                  0                               405.237462                              257.299895                                   66.21                                 2.286086                                  847.63                                                 29                              219.021420                                11.174282                                5.424407                                 0.429374                               10.724241                                           0.408248                           2.428992                                 1477.97                                82.109444                               50.110120                                   20.06                                 0.854976                                                  5                                 18                                 2.246479                                  0.678544                                -0.245703                                 1.000771                                -0.941078                                          -2.449490                            -1.507217                                  889.21                                55.579412                               35.704680                                -0.289466                                  126.74                                                  4                                 11                              1039.436667                                 67.539577                                42.883316                                11.035000                                  0.381014                               141.271667                                           4.833333                            15.500000                                         1                                                  4                                         1                                          1                                            1                                          1                                              0                                                 0                                                 1                                                 0                                                  0                                            1                                             0                                        1                                               0                                          1                                                 0                                              1                                         3                                            0                                            0                                            1                                            0                                            0                                                  0                                             0                                             1                                              0
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                    0                    1                    0                   0                          0                        1                        0                        0                        0                              0                       0                       1                       0                           0                           0                           0                           0                           1                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                               548.905851                              258.700528                                  154.60                                -0.277640                                  931.63                                                 35                              251.609234                                11.477071                               15.874374                                 0.509798                               17.221593                                           0.000000                           3.450328                                 1320.64                                96.581000                               47.935920                                   56.46                                 0.755711                                                  5                                 18                                -0.440929                                  0.235296                                 0.013087                                 2.154929                                -1.539467                                           0.000000                            -0.303276                                  634.84                                61.910000                               27.839228                                -0.763603                                  100.04                                                  5                                  8                              1028.611429                                 78.415122                                36.957218                                22.085714                                 -0.039663                               133.090000                                           5.000000                            13.285714                                         1                                                  4                                         1                                          1                                            1                                          1                                              0                                                 1                                                 0                                                 0                                                  0                                            1                                             0                                        1                                               0                                          1                                                 0                                              1                                         3                                            0                                            0                                            0                                            1                                            0                                                  0                                             0                                             1                                              0

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SUM(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.COUNT(transactions))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.COUNT(transactions))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.COUNT(transactions))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.COUNT(transactions))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.COUNT(transactions))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                               524.919674                              217.064024                                  198.92                                -1.546156                                  834.08                                                 25                              327.331994                                25.801702                               45.471977                                 0.326628                               11.742343                                           1.329160                           5.741661                                 1418.99                               136.672500                               51.192518                                  124.58                                 0.164928                                                  5                                 19                                 0.130163                                  1.796000                                -1.336611                                 2.290693                                -1.165955                                          -1.206607                            -0.243198                                  546.69                                64.238824                                9.176046                                -0.614442                                  119.86                                                  2                                  4                               962.735000                                 87.486612                                36.177337                                33.153333                                 -0.257693                               139.013333                                           4.166667                            12.166667                                         1                                                  4                                         1                                          1                                            1                                       True                                          False                                             False                                             False                                              True                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                               733.862898                              319.497611                                  193.08                                -1.797214                                 1180.90                                                 43                              333.923377                                11.875823                               17.851716                                 0.324894                               11.457114                                           0.440959                           4.272002                                 1650.65                               104.565000                               41.627134                                   60.29                                 0.417250                                                  5                                 21                                -0.385392                                  1.078619                                -1.743267                                 1.490781                                 0.230847                                          -1.619848                            -0.086578                                  557.32                                69.665000                               22.026552                                -0.624344                                  118.59                                                  4                                  8                              1130.936667                                 81.540322                                35.499735                                21.453333                                 -0.199690                               131.211111                                           4.777778                            14.000000                                         1                                                  5                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                               407.390549                              215.417211                                  108.69                                 0.353061                                  715.80                                                 23                              450.318768                                10.012657                               10.829338                                 0.530094                                5.596950                                           0.547723                           4.868265                                 1691.27                                92.004000                               46.402508                                   38.69                                 0.785376                                                  5                                 19                                 0.727969                                 -0.730554                                -0.627238                                 1.010658                                -1.598919                                          -0.608581                             0.570300                                  653.11                                66.563000                               37.945636                                -0.608419                                  133.75                                                  4                                  8                              1051.790000                                 81.478110                                43.083442                                21.738000                                  0.070612                               143.160000                                           4.600000                            12.800000                                         1                                                  2                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                               615.714934                              315.817331                                  136.01                                -0.082021                                 1100.82                                                 39                              346.152626                                10.757169                               10.517928                                 0.305345                                9.491736                                           0.353553                           4.086126                                 1690.97                                87.669412                               43.950396                                   40.88                                 0.454842                                                  5                                 21                                -0.576079                                 -2.048650                                -0.403715                                 1.981423                                -0.964539                                          -2.828427                            -1.158217                                  619.93                                52.288421                               33.618728                                -0.522578                                  120.06                                                  4                                  8                              1234.708750                                 76.964367                                39.477166                                17.001250                                 -0.010253                               137.602500                                           4.875000                            16.125000                                         1                                                  4                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                               584.302915                              261.535265                                  137.50                                -0.595128                                  997.48                                                 33                              263.424967                                 7.689676                               10.431664                                 0.440504                                5.679896                                           0.487950                           2.699206                                 1705.25                                95.488000                               44.978308                                   34.93                                 0.538086                                                  5                                 19                                 0.215180                                  0.246613                                -0.405840                                 0.013052                                -0.308063                                          -1.229634                            -0.566615                                  921.73                                73.576471                               26.724118                                -0.618136                                  133.41                                                  4                                 11                              1288.391429                                 83.471845                                37.362181                                19.642857                                 -0.085018                               142.497143                                           4.714286                            15.428571                                         1                                                  4                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SUM(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.COUNT(transactions))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.COUNT(transactions))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.COUNT(transactions))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.COUNT(transactions))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.COUNT(transactions))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                               524.919674                              217.064024                                  198.92                                -1.546156                                  834.08                                                 25                              327.331994                                25.801702                               45.471977                                 0.326628                               11.742343                                           1.329160                           5.741661                                 1418.99                               136.672500                               51.192518                                  124.58                                 0.164928                                                  5                                 19                                 0.130163                                  1.796000                                -1.336611                                 2.290693                                -1.165955                                          -1.206607                            -0.243198                                  546.69                                64.238824                                9.176046                                -0.614442                                  119.86                                                  2                                  4                               962.735000                                 87.486612                                36.177337                                33.153333                                 -0.257693                               139.013333                                           4.166667                            12.166667                                         1                                                  4                                         1                                          1                                            1                                       True                                          False                                             False                                             False                                              True                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                               733.862898                              319.497611                                  193.08                                -1.797214                                 1180.90                                                 43                              333.923377                                11.875823                               17.851716                                 0.324894                               11.457114                                           0.440959                           4.272002                                 1650.65                               104.565000                               41.627134                                   60.29                                 0.417250                                                  5                                 21                                -0.385392                                  1.078619                                -1.743267                                 1.490781                                 0.230847                                          -1.619848                            -0.086578                                  557.32                                69.665000                               22.026552                                -0.624344                                  118.59                                                  4                                  8                              1130.936667                                 81.540322                                35.499735                                21.453333                                 -0.199690                               131.211111                                           4.777778                            14.000000                                         1                                                  5                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                               407.390549                              215.417211                                  108.69                                 0.353061                                  715.80                                                 23                              450.318768                                10.012657                               10.829338                                 0.530094                                5.596950                                           0.547723                           4.868265                                 1691.27                                92.004000                               46.402508                                   38.69                                 0.785376                                                  5                                 19                                 0.727969                                 -0.730554                                -0.627238                                 1.010658                                -1.598919                                          -0.608581                             0.570300                                  653.11                                66.563000                               37.945636                                -0.608419                                  133.75                                                  4                                  8                              1051.790000                                 81.478110                                43.083442                                21.738000                                  0.070612                               143.160000                                           4.600000                            12.800000                                         1                                                  2                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                               615.714934                              315.817331                                  136.01                                -0.082021                                 1100.82                                                 39                              346.152626                                10.757169                               10.517928                                 0.305345                                9.491736                                           0.353553                           4.086126                                 1690.97                                87.669412                               43.950396                                   40.88                                 0.454842                                                  5                                 21                                -0.576079                                 -2.048650                                -0.403715                                 1.981423                                -0.964539                                          -2.828427                            -1.158217                                  619.93                                52.288421                               33.618728                                -0.522578                                  120.06                                                  4                                  8                              1234.708750                                 76.964367                                39.477166                                17.001250                                 -0.010253                               137.602500                                           4.875000                            16.125000                                         1                                                  4                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                               584.302915                              261.535265                                  137.50                                -0.595128                                  997.48                                                 33                              263.424967                                 7.689676                               10.431664                                 0.440504                                5.679896                                           0.487950                           2.699206                                 1705.25                                95.488000                               44.978308                                   34.93                                 0.538086                                                  5                                 19                                 0.215180                                  0.246613                                -0.405840                                 0.013052                                -0.308063                                          -1.229634                            -0.566615                                  921.73                                73.576471                               26.724118                                -0.618136                                  133.41                                                  4                                 11                              1288.391429                                 83.471845                                37.362181                                19.642857                                 -0.085018                               142.497143                                           4.714286                            15.428571                                         1                                                  4                                         1                                          1                                            1                                       True                                          False                                             False                                              True                                             False                                              False                                         True                                         False                                     True                                           False                                       True                                             False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False