Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  SUM(sessions.STD(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SUM(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.COUNT(transactions))  MAX(sessions.STD(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.COUNT(transactions))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.COUNT(transactions))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.COUNT(transactions))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                        0                        1                        0                        0                              0                    1                    0                    0                   0                          0                           0                           0                           0                           0                           1                               0                       0                       0                       1                           0                         0                         1                          0                                0                     0                     0                     1                            0                           0                           1                           0                           0                                  0                       0                       1                       0                              0                              259.873954                                 0.014384                                                 30                               472.231119                                   86.49                                  839.76                                 0.415426                                           0.000000                              402.775486                                11.007471                                4.961414                                7.928001                           3.600926                               51.149250                                 0.602209                                                  5                                 1700.67                                94.481667                                   20.65                                 18                                 0.204548                                           0.000000                                  0.472342                                  0.335175                                -0.470410                                -0.333796                           -0.317685                               36.734681                                -0.539060                                                  5                                  543.18                                66.666667                                  128.51                                  8                                43.312326                                  0.002397                                           5.000000                               1058.276667                                 78.705187                                14.415000                               139.960000                           13.166667                                            1                                          1                                         1                                        1                                                  5                                           1                                                 0                                        1                                               0                                          1                                              0                                      1                                             0                                                 1                                                 0                                                 0                                                  0                                                    1                                         3                                            1                                            0                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                        0                        0                        0                        1                              0                    0                    0                    0                   1                          0                           1                           0                           0                           0                           0                               0                       1                       0                       0                           0                         1                         0                          0                                0                     1                     0                     0                            0                           0                           0                           0                           1                                  0                       0                       0                       1                              0                              356.125829                                 0.002764                                                 37                               649.657515                                  131.51                                 1157.99                                 0.387884                                           0.517549                              235.992478                                13.027258                               16.960575                                3.514421                           3.335416                               54.293903                                 0.382868                                                  5                                 1351.46                               110.450000                                   54.83                                 18                                -1.065663                                          -0.644061                                 -0.391805                                  1.980948                                 2.103510                                 0.027256                            0.282488                               29.026424                                -0.711744                                                  4                                  771.68                                70.638182                                  139.20                                 10                                44.515729                                  0.000346                                           4.625000                               1090.960000                                 81.207189                                16.438750                               144.748750                           13.625000                                            1                                          1                                         1                                        1                                                  5                                           1                                                 0                                        1                                               0                                          1                                              0                                      1                                             0                                                 0                                                 1                                                 0                                                  0                                                    1                                         3                                            0                                            1                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                        1                        0                        0                        0                              0                    1                    0                    0                   0                          0                           0                           0                           1                           0                           0                               0                       1                       0                       0                           0                         0                         1                          0                                0                     1                     0                     0                            0                           1                           0                           0                           0                                  0                       1                       0                       0                              0                              312.745952                                -0.476122                                                 40                               582.193117                                   78.59                                 1057.97                                 0.589386                                           0.000000                              279.510713                                13.759314                                6.954507                                7.322191                           4.062019                               46.905665                                 0.640252                                                  5                                 1613.93                                88.755625                                   26.36                                 25                                -0.312355                                           0.000000                                  0.778170                                 -0.424949                                 2.440005                                -0.780493                            1.946018                               30.450261                                -1.038434                                                  5                                  809.97                                50.623125                                  118.90                                 12                                39.093244                                 -0.059515                                           5.000000                               1128.202500                                 72.774140                                 9.823750                               132.246250                           15.750000                                            1                                          1                                         1                                        1                                                  4                                           1                                                 0                                        1                                               0                                          1                                              0                                      1                                             0                                                 0                                                 0                                                 1                                                  0                                                    1                                         3                                            0                                            0                                            0                                            0                                            1                                                  0                                             1                                             0                                              0
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                        0                        0                        1                        0                              0                    0                    0                    1                   0                          0                           0                           1                           0                           0                           0                               0                       1                       0                       0                           0                         0                         0                          1                                0                     0                     1                     0                            0                           0                           0                           1                           0                                  0                       0                       1                       0                              0                              257.299895                                 2.286086                                                 29                               405.237462                                   66.21                                  847.63                                 0.429374                                           0.408248                              219.021420                                11.174282                                5.424407                               10.724241                           2.428992                               50.110120                                 0.854976                                                  5                                 1477.97                                82.109444                                   20.06                                 18                                -0.245703                                          -2.449490                                  2.246479                                  0.678544                                 1.000771                                -0.941078                           -1.507217                               35.704680                                -0.289466                                                  4                                  889.21                                55.579412                                  126.74                                 11                                42.883316                                  0.381014                                           4.833333                               1039.436667                                 67.539577                                11.035000                               141.271667                           15.500000                                            1                                          1                                         1                                        1                                                  4                                           1                                                 0                                        1                                               0                                          1                                              0                                      1                                             0                                                 0                                                 1                                                 0                                                  0                                                    1                                         3                                            0                                            0                                            1                                            0                                            0                                                  0                                             0                                             1                                              0
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                        1                        0                        0                        0                              0                    0                    1                    0                   0                          0                           0                           0                           0                           1                           0                               0                       0                       1                       0                           0                         1                         0                          0                                0                     1                     0                     0                            0                           1                           0                           0                           0                                  0                       1                       0                       0                              0                              258.700528                                -0.277640                                                 35                               548.905851                                  154.60                                  931.63                                 0.509798                                           0.000000                              251.609234                                11.477071                               15.874374                               17.221593                           3.450328                               47.935920                                 0.755711                                                  5                                 1320.64                                96.581000                                   56.46                                 18                                 0.013087                                           0.000000                                 -0.440929                                  0.235296                                 2.154929                                -1.539467                           -0.303276                               27.839228                                -0.763603                                                  5                                  634.84                                61.910000                                  100.04                                  8                                36.957218                                 -0.039663                                           5.000000                               1028.611429                                 78.415122                                22.085714                               133.090000                           13.285714                                            1                                          1                                         1                                        1                                                  4                                           1                                                 0                                        1                                               0                                          1                                              0                                      1                                             0                                                 1                                                 0                                                 0                                                  0                                                    1                                         3                                            0                                            0                                            0                                            1                                            0                                                  0                                             0                                             1                                              0

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  SUM(sessions.STD(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SUM(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.COUNT(transactions))  MAX(sessions.STD(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.COUNT(transactions))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.COUNT(transactions))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.COUNT(transactions))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                              217.064024                                -1.546156                                                 25                               524.919674                                  198.92                                  834.08                                 0.326628                                           1.329160                              327.331994                                25.801702                               45.471977                               11.742343                           5.741661                               51.192518                                 0.164928                                                  5                                 1418.99                               136.672500                                  124.58                                 19                                -1.336611                                          -1.206607                                  0.130163                                  1.796000                                 2.290693                                -1.165955                           -0.243198                                9.176046                                -0.614442                                                  2                                  546.69                                64.238824                                  119.86                                  4                                36.177337                                 -0.257693                                           4.166667                                962.735000                                 87.486612                                33.153333                               139.013333                           12.166667                                            1                                          1                                         1                                        1                                                  4                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                             False                                              True                                              False                                                    1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                       True                            False                 False                 False                  True                        False                       False                       False                        True                       False                              False                   False                   False                    True                          False                              319.497611                                -1.797214                                                 43                               733.862898                                  193.08                                 1180.90                                 0.324894                                           0.440959                              333.923377                                11.875823                               17.851716                               11.457114                           4.272002                               41.627134                                 0.417250                                                  5                                 1650.65                               104.565000                                   60.29                                 21                                -1.743267                                          -1.619848                                 -0.385392                                  1.078619                                 1.490781                                 0.230847                           -0.086578                               22.026552                                -0.624344                                                  4                                  557.32                                69.665000                                  118.59                                  8                                35.499735                                 -0.199690                                           4.777778                               1130.936667                                 81.540322                                21.453333                               131.211111                           14.000000                                            1                                          1                                         1                                        1                                                  5                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                        True                       False                       False                       False                       False                           False                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                   False                    True                   False                          False                              215.417211                                 0.353061                                                 23                               407.390549                                  108.69                                  715.80                                 0.530094                                           0.547723                              450.318768                                10.012657                               10.829338                                5.596950                           4.868265                               46.402508                                 0.785376                                                  5                                 1691.27                                92.004000                                   38.69                                 19                                -0.627238                                          -0.608581                                  0.727969                                 -0.730554                                 1.010658                                -1.598919                            0.570300                               37.945636                                -0.608419                                                  4                                  653.11                                66.563000                                  133.75                                  8                                43.083442                                  0.070612                                           4.600000                               1051.790000                                 81.478110                                21.738000                               143.160000                           12.800000                                            1                                          1                                         1                                        1                                                  2                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                              315.817331                                -0.082021                                                 39                               615.714934                                  136.01                                 1100.82                                 0.305345                                           0.353553                              346.152626                                10.757169                               10.517928                                9.491736                           4.086126                               43.950396                                 0.454842                                                  5                                 1690.97                                87.669412                                   40.88                                 21                                -0.403715                                          -2.828427                                 -0.576079                                 -2.048650                                 1.981423                                -0.964539                           -1.158217                               33.618728                                -0.522578                                                  4                                  619.93                                52.288421                                  120.06                                  8                                39.477166                                 -0.010253                                           4.875000                               1234.708750                                 76.964367                                17.001250                               137.602500                           16.125000                                            1                                          1                                         1                                        1                                                  4                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                    False                     True                    False                    False                          False                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                        True                              False                   False                   False                    True                          False                              261.535265                                -0.595128                                                 33                               584.302915                                  137.50                                  997.48                                 0.440504                                           0.487950                              263.424967                                 7.689676                               10.431664                                5.679896                           2.699206                               44.978308                                 0.538086                                                  5                                 1705.25                                95.488000                                   34.93                                 19                                -0.405840                                          -1.229634                                  0.215180                                  0.246613                                 0.013052                                -0.308063                           -0.566615                               26.724118                                -0.618136                                                  4                                  921.73                                73.576471                                  133.41                                 11                                37.362181                                 -0.085018                                           4.714286                               1288.391429                                 83.471845                                19.642857                               142.497143                           15.428571                                            1                                          1                                         1                                        1                                                  4                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  SUM(sessions.STD(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SUM(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.MAX(transactions.amount))  STD(sessions.COUNT(transactions))  MAX(sessions.STD(transactions.amount))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.COUNT(transactions))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.COUNT(transactions))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.COUNT(transactions))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.COUNT(transactions))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                              217.064024                                -1.546156                                                 25                               524.919674                                  198.92                                  834.08                                 0.326628                                           1.329160                              327.331994                                25.801702                               45.471977                               11.742343                           5.741661                               51.192518                                 0.164928                                                  5                                 1418.99                               136.672500                                  124.58                                 19                                -1.336611                                          -1.206607                                  0.130163                                  1.796000                                 2.290693                                -1.165955                           -0.243198                                9.176046                                -0.614442                                                  2                                  546.69                                64.238824                                  119.86                                  4                                36.177337                                 -0.257693                                           4.166667                                962.735000                                 87.486612                                33.153333                               139.013333                           12.166667                                            1                                          1                                         1                                        1                                                  4                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                             False                                              True                                              False                                                    1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                       True                            False                 False                 False                  True                        False                       False                       False                        True                       False                              False                   False                   False                    True                          False                              319.497611                                -1.797214                                                 43                               733.862898                                  193.08                                 1180.90                                 0.324894                                           0.440959                              333.923377                                11.875823                               17.851716                               11.457114                           4.272002                               41.627134                                 0.417250                                                  5                                 1650.65                               104.565000                                   60.29                                 21                                -1.743267                                          -1.619848                                 -0.385392                                  1.078619                                 1.490781                                 0.230847                           -0.086578                               22.026552                                -0.624344                                                  4                                  557.32                                69.665000                                  118.59                                  8                                35.499735                                 -0.199690                                           4.777778                               1130.936667                                 81.540322                                21.453333                               131.211111                           14.000000                                            1                                          1                                         1                                        1                                                  5                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                        True                       False                       False                       False                       False                           False                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                   False                    True                   False                          False                              215.417211                                 0.353061                                                 23                               407.390549                                  108.69                                  715.80                                 0.530094                                           0.547723                              450.318768                                10.012657                               10.829338                                5.596950                           4.868265                               46.402508                                 0.785376                                                  5                                 1691.27                                92.004000                                   38.69                                 19                                -0.627238                                          -0.608581                                  0.727969                                 -0.730554                                 1.010658                                -1.598919                            0.570300                               37.945636                                -0.608419                                                  4                                  653.11                                66.563000                                  133.75                                  8                                43.083442                                  0.070612                                           4.600000                               1051.790000                                 81.478110                                21.738000                               143.160000                           12.800000                                            1                                          1                                         1                                        1                                                  2                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                              315.817331                                -0.082021                                                 39                               615.714934                                  136.01                                 1100.82                                 0.305345                                           0.353553                              346.152626                                10.757169                               10.517928                                9.491736                           4.086126                               43.950396                                 0.454842                                                  5                                 1690.97                                87.669412                                   40.88                                 21                                -0.403715                                          -2.828427                                 -0.576079                                 -2.048650                                 1.981423                                -0.964539                           -1.158217                               33.618728                                -0.522578                                                  4                                  619.93                                52.288421                                  120.06                                  8                                39.477166                                 -0.010253                                           4.875000                               1234.708750                                 76.964367                                17.001250                               137.602500                           16.125000                                            1                                          1                                         1                                        1                                                  4                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                    False                     True                    False                    False                          False                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                        True                              False                   False                   False                    True                          False                              261.535265                                -0.595128                                                 33                               584.302915                                  137.50                                  997.48                                 0.440504                                           0.487950                              263.424967                                 7.689676                               10.431664                                5.679896                           2.699206                               44.978308                                 0.538086                                                  5                                 1705.25                                95.488000                                   34.93                                 19                                -0.405840                                          -1.229634                                  0.215180                                  0.246613                                 0.013052                                -0.308063                           -0.566615                               26.724118                                -0.618136                                                  4                                  921.73                                73.576471                                  133.41                                 11                                37.362181                                 -0.085018                                           4.714286                               1288.391429                                 83.471845                                19.642857                               142.497143                           15.428571                                            1                                          1                                         1                                        1                                                  4                                        True                                             False                                     True                                           False                                       True                                          False                                   True                                         False                                             False                                              True                                             False                                              False                                                    1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False