Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.MIN(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                    1                    0                    0                   0                          0                        0                        1                        0                        0                              0                       0                       0                       1                           0                           0                           0                           0                           0                           1                               0                     0                     0                     1                            0                         0                         1                          0                                0                       0                       1                       0                              0                           0                           1                           0                           0                                  0                                   86.49                               472.231119                                 0.014384                              259.873954                                  839.76                                                 30                                4.961414                              402.775486                           3.600926                                11.007471                                 0.415426                                7.928001                                           0.000000                                   20.65                                 1700.67                                 18                                94.481667                                 0.602209                               51.149250                                                  5                                -0.470410                                 0.472342                           -0.317685                                  0.335175                                 0.204548                                -0.333796                                           0.000000                                   543.18                                  8                                66.666667                                -0.539060                               36.734681                                  128.51                                                  5                                14.415000                              1058.276667                           13.166667                                 78.705187                                  0.002397                                43.312326                               139.960000                                           5.000000                                             1                                          1                                                  5                                          1                                        1                                          1                                                 0                                        1                                               0                                                 1                                                 0                                                 0                                                  0                                                1                                              0                                      1                                             0                                              1                                         3                                            1                                            0                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                    0                    0                    0                   1                          0                        0                        0                        0                        1                              0                       1                       0                       0                           0                           1                           0                           0                           0                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       0                       0                       1                              0                           0                           0                           0                           1                                  0                                  131.51                               649.657515                                 0.002764                              356.125829                                 1157.99                                                 37                               16.960575                              235.992478                           3.335416                                13.027258                                 0.387884                                3.514421                                           0.517549                                   54.83                                 1351.46                                 18                               110.450000                                 0.382868                               54.293903                                                  5                                 2.103510                                -0.391805                            0.282488                                  1.980948                                -1.065663                                 0.027256                                          -0.644061                                   771.68                                 10                                70.638182                                -0.711744                               29.026424                                  139.20                                                  4                                16.438750                              1090.960000                           13.625000                                 81.207189                                  0.000346                                44.515729                               144.748750                                           4.625000                                             1                                          1                                                  5                                          1                                        1                                          1                                                 0                                        1                                               0                                                 0                                                 1                                                 0                                                  0                                                1                                              0                                      1                                             0                                              1                                         3                                            0                                            1                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                    1                    0                    0                   0                          0                        1                        0                        0                        0                              0                       1                       0                       0                           0                           0                           0                           1                           0                           0                               0                     1                     0                     0                            0                         0                         1                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                   78.59                               582.193117                                -0.476122                              312.745952                                 1057.97                                                 40                                6.954507                              279.510713                           4.062019                                13.759314                                 0.589386                                7.322191                                           0.000000                                   26.36                                 1613.93                                 25                                88.755625                                 0.640252                               46.905665                                                  5                                 2.440005                                 0.778170                            1.946018                                 -0.424949                                -0.312355                                -0.780493                                           0.000000                                   809.97                                 12                                50.623125                                -1.038434                               30.450261                                  118.90                                                  5                                 9.823750                              1128.202500                           15.750000                                 72.774140                                 -0.059515                                39.093244                               132.246250                                           5.000000                                             1                                          1                                                  4                                          1                                        1                                          1                                                 0                                        1                                               0                                                 0                                                 0                                                 1                                                  0                                                1                                              0                                      1                                             0                                              1                                         3                                            0                                            0                                            0                                            0                                            1                                                  0                                             1                                             0                                              0
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                    0                    0                    1                   0                          0                        0                        0                        1                        0                              0                       1                       0                       0                           0                           0                           1                           0                           0                           0                               0                     0                     1                     0                            0                         0                         0                          1                                0                       0                       1                       0                              0                           0                           0                           1                           0                                  0                                   66.21                               405.237462                                 2.286086                              257.299895                                  847.63                                                 29                                5.424407                              219.021420                           2.428992                                11.174282                                 0.429374                               10.724241                                           0.408248                                   20.06                                 1477.97                                 18                                82.109444                                 0.854976                               50.110120                                                  5                                 1.000771                                 2.246479                           -1.507217                                  0.678544                                -0.245703                                -0.941078                                          -2.449490                                   889.21                                 11                                55.579412                                -0.289466                               35.704680                                  126.74                                                  4                                11.035000                              1039.436667                           15.500000                                 67.539577                                  0.381014                                42.883316                               141.271667                                           4.833333                                             1                                          1                                                  4                                          1                                        1                                          1                                                 0                                        1                                               0                                                 0                                                 1                                                 0                                                  0                                                1                                              0                                      1                                             0                                              1                                         3                                            0                                            0                                            1                                            0                                            0                                                  0                                             0                                             1                                              0
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                    0                    1                    0                   0                          0                        1                        0                        0                        0                              0                       0                       1                       0                           0                           0                           0                           0                           1                           0                               0                     1                     0                     0                            0                         1                         0                          0                                0                       1                       0                       0                              0                           1                           0                           0                           0                                  0                                  154.60                               548.905851                                -0.277640                              258.700528                                  931.63                                                 35                               15.874374                              251.609234                           3.450328                                11.477071                                 0.509798                               17.221593                                           0.000000                                   56.46                                 1320.64                                 18                                96.581000                                 0.755711                               47.935920                                                  5                                 2.154929                                -0.440929                           -0.303276                                  0.235296                                 0.013087                                -1.539467                                           0.000000                                   634.84                                  8                                61.910000                                -0.763603                               27.839228                                  100.04                                                  5                                22.085714                              1028.611429                           13.285714                                 78.415122                                 -0.039663                                36.957218                               133.090000                                           5.000000                                             1                                          1                                                  4                                          1                                        1                                          1                                                 0                                        1                                               0                                                 1                                                 0                                                 0                                                  0                                                1                                              0                                      1                                             0                                              1                                         3                                            0                                            0                                            0                                            1                                            0                                                  0                                             0                                             1                                              0

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.MIN(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                  198.92                               524.919674                                -1.546156                              217.064024                                  834.08                                                 25                               45.471977                              327.331994                           5.741661                                25.801702                                 0.326628                               11.742343                                           1.329160                                  124.58                                 1418.99                                 19                               136.672500                                 0.164928                               51.192518                                                  5                                 2.290693                                 0.130163                           -0.243198                                  1.796000                                -1.336611                                -1.165955                                          -1.206607                                   546.69                                  4                                64.238824                                -0.614442                                9.176046                                  119.86                                                  2                                33.153333                               962.735000                           12.166667                                 87.486612                                 -0.257693                                36.177337                               139.013333                                           4.166667                                             1                                          1                                                  4                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                             False                                              True                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                  193.08                               733.862898                                -1.797214                              319.497611                                 1180.90                                                 43                               17.851716                              333.923377                           4.272002                                11.875823                                 0.324894                               11.457114                                           0.440959                                   60.29                                 1650.65                                 21                               104.565000                                 0.417250                               41.627134                                                  5                                 1.490781                                -0.385392                           -0.086578                                  1.078619                                -1.743267                                 0.230847                                          -1.619848                                   557.32                                  8                                69.665000                                -0.624344                               22.026552                                  118.59                                                  4                                21.453333                              1130.936667                           14.000000                                 81.540322                                 -0.199690                                35.499735                               131.211111                                           4.777778                                             1                                          1                                                  5                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                  108.69                               407.390549                                 0.353061                              215.417211                                  715.80                                                 23                               10.829338                              450.318768                           4.868265                                10.012657                                 0.530094                                5.596950                                           0.547723                                   38.69                                 1691.27                                 19                                92.004000                                 0.785376                               46.402508                                                  5                                 1.010658                                 0.727969                            0.570300                                 -0.730554                                -0.627238                                -1.598919                                          -0.608581                                   653.11                                  8                                66.563000                                -0.608419                               37.945636                                  133.75                                                  4                                21.738000                              1051.790000                           12.800000                                 81.478110                                  0.070612                                43.083442                               143.160000                                           4.600000                                             1                                          1                                                  2                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                  136.01                               615.714934                                -0.082021                              315.817331                                 1100.82                                                 39                               10.517928                              346.152626                           4.086126                                10.757169                                 0.305345                                9.491736                                           0.353553                                   40.88                                 1690.97                                 21                                87.669412                                 0.454842                               43.950396                                                  5                                 1.981423                                -0.576079                           -1.158217                                 -2.048650                                -0.403715                                -0.964539                                          -2.828427                                   619.93                                  8                                52.288421                                -0.522578                               33.618728                                  120.06                                                  4                                17.001250                              1234.708750                           16.125000                                 76.964367                                 -0.010253                                39.477166                               137.602500                                           4.875000                                             1                                          1                                                  4                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                  137.50                               584.302915                                -0.595128                              261.535265                                  997.48                                                 33                               10.431664                              263.424967                           2.699206                                 7.689676                                 0.440504                                5.679896                                           0.487950                                   34.93                                 1705.25                                 19                                95.488000                                 0.538086                               44.978308                                                  5                                 0.013052                                 0.215180                           -0.566615                                  0.246613                                -0.405840                                -0.308063                                          -1.229634                                   921.73                                 11                                73.576471                                -0.618136                               26.724118                                  133.41                                                  4                                19.642857                              1288.391429                           15.428571                                 83.471845                                 -0.085018                                37.362181                               142.497143                                           4.714286                                             1                                          1                                                  4                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  SUM(sessions.MIN(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MEAN(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                  198.92                               524.919674                                -1.546156                              217.064024                                  834.08                                                 25                               45.471977                              327.331994                           5.741661                                25.801702                                 0.326628                               11.742343                                           1.329160                                  124.58                                 1418.99                                 19                               136.672500                                 0.164928                               51.192518                                                  5                                 2.290693                                 0.130163                           -0.243198                                  1.796000                                -1.336611                                -1.165955                                          -1.206607                                   546.69                                  4                                64.238824                                -0.614442                                9.176046                                  119.86                                                  2                                33.153333                               962.735000                           12.166667                                 87.486612                                 -0.257693                                36.177337                               139.013333                                           4.166667                                             1                                          1                                                  4                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                             False                                              True                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                  True                        False                     False                     False                       True                            False                   False                   False                    True                          False                       False                       False                        True                       False                              False                                  193.08                               733.862898                                -1.797214                              319.497611                                 1180.90                                                 43                               17.851716                              333.923377                           4.272002                                11.875823                                 0.324894                               11.457114                                           0.440959                                   60.29                                 1650.65                                 21                               104.565000                                 0.417250                               41.627134                                                  5                                 1.490781                                -0.385392                           -0.086578                                  1.078619                                -1.743267                                 0.230847                                          -1.619848                                   557.32                                  8                                69.665000                                -0.624344                               22.026552                                  118.59                                                  4                                21.453333                              1130.936667                           14.000000                                 81.540322                                 -0.199690                                35.499735                               131.211111                                           4.777778                                             1                                          1                                                  5                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                        True                       False                       False                       False                       False                           False                 False                 False                 False                         True                     False                     False                      False                             True                   False                    True                   False                          False                       False                       False                       False                       False                               True                                  108.69                               407.390549                                 0.353061                              215.417211                                  715.80                                                 23                               10.829338                              450.318768                           4.868265                                10.012657                                 0.530094                                5.596950                                           0.547723                                   38.69                                 1691.27                                 19                                92.004000                                 0.785376                               46.402508                                                  5                                 1.010658                                 0.727969                            0.570300                                 -0.730554                                -0.627238                                -1.598919                                          -0.608581                                   653.11                                  8                                66.563000                                -0.608419                               37.945636                                  133.75                                                  4                                21.738000                              1051.790000                           12.800000                                 81.478110                                  0.070612                                43.083442                               143.160000                                           4.600000                                             1                                          1                                                  2                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                False                False                False               False                       True                    False                    False                    False                    False                           True                   False                   False                    True                       False                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                    True                   False                   False                          False                       False                       False                       False                       False                               True                                  136.01                               615.714934                                -0.082021                              315.817331                                 1100.82                                                 39                               10.517928                              346.152626                           4.086126                                10.757169                                 0.305345                                9.491736                                           0.353553                                   40.88                                 1690.97                                 21                                87.669412                                 0.454842                               43.950396                                                  5                                 1.981423                                -0.576079                           -1.158217                                 -2.048650                                -0.403715                                -0.964539                                          -2.828427                                   619.93                                  8                                52.288421                                -0.522578                               33.618728                                  120.06                                                  4                                17.001250                              1234.708750                           16.125000                                 76.964367                                 -0.010253                                39.477166                               137.602500                                           4.875000                                             1                                          1                                                  4                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                False                False                False               False                       True                    False                     True                    False                    False                          False                   False                   False                   False                        True                       False                       False                       False                       False                       False                            True                 False                 False                 False                         True                     False                     False                      False                             True                   False                   False                    True                          False                       False                       False                       False                        True                              False                                  137.50                               584.302915                                -0.595128                              261.535265                                  997.48                                                 33                               10.431664                              263.424967                           2.699206                                 7.689676                                 0.440504                                5.679896                                           0.487950                                   34.93                                 1705.25                                 19                                95.488000                                 0.538086                               44.978308                                                  5                                 0.013052                                 0.215180                           -0.566615                                  0.246613                                -0.405840                                -0.308063                                          -1.229634                                   921.73                                 11                                73.576471                                -0.618136                               26.724118                                  133.41                                                  4                                19.642857                              1288.391429                           15.428571                                 83.471845                                 -0.085018                                37.362181                               142.497143                                           4.714286                                             1                                          1                                                  4                                          1                                        1                                       True                                             False                                     True                                           False                                             False                                              True                                             False                                              False                                             True                                          False                                   True                                         False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False