Problem Statement¶
Summary of this page: This project is about a company trying to identify its customer segments and give actionable insights into the characteristics of each. I began by defining the problem and loading the dataset into Python for analysis. I explored the data through descriptive statistics and visualizations, then cleaned and transformed it to prepare for modeling. From there, I tested multiple approaches (e.g., similarity-based vs. factorization), tuned parameters, compared their performance, and evaluated the results against the original problem goals. The project concludes with insights and recommendations that highlight both the strengths and limitations of the models.
[ Jump to Conclusion and Business Application ]
Business Context¶
Understanding customer personality and behavior is pivotal for businesses to enhance customer satisfaction and increase revenue. Segmentation based on a customer's personality, demographics, and purchasing behavior allows companies to create tailored marketing campaigns, improve customer retention, and optimize product offerings.
A leading retail company with a rapidly growing customer base seeks to gain deeper insights into their customers' profiles. The company recognizes that understanding customer personalities, lifestyles, and purchasing habits can unlock significant opportunities for personalizing marketing strategies and creating loyalty programs. These insights can help address critical business challenges, such as improving the effectiveness of marketing campaigns, identifying high-value customer groups, and fostering long-term relationships with customers.
With the competition intensifying in the retail space, moving away from generic strategies to more targeted and personalized approaches is essential for sustaining a competitive edge.
Objective¶
In an effort to optimize marketing efficiency and enhance customer experience, the company has embarked on a mission to identify distinct customer segments. By understanding the characteristics, preferences, and behaviors of each group, the company aims to:
- Develop personalized marketing campaigns to increase conversion rates.
- Create effective retention strategies for high-value customers.
- Optimize resource allocation, such as inventory management, pricing strategies, and store layouts.
As a data scientist tasked with this project, your responsibility is to analyze the given customer data, apply machine learning techniques to segment the customer base, and provide actionable insights into the characteristics of each segment.
Let us start tackling this problem.
Data Dictionary¶
The dataset includes historical data on customer demographics, personality traits, and purchasing behaviors. Key attributes are:
Customer Information
- ID: Unique identifier for each customer.
- Year_Birth: Customer's year of birth.
- Education: Education level of the customer.
- Marital_Status: Marital status of the customer.
- Income: Yearly household income (in dollars).
- Kidhome: Number of children in the household.
- Teenhome: Number of teenagers in the household.
- Dt_Customer: Date when the customer enrolled with the company.
- Recency: Number of days since the customer’s last purchase.
- Complain: Whether the customer complained in the last 2 years (1 for yes, 0 for no).
Spending Information (Last 2 Years)
- MntWines: Amount spent on wine.
- MntFruits: Amount spent on fruits.
- MntMeatProducts: Amount spent on meat.
- MntFishProducts: Amount spent on fish.
- MntSweetProducts: Amount spent on sweets.
- MntGoldProds: Amount spent on gold products.
Purchase and Campaign Interaction
- NumDealsPurchases: Number of purchases made using a discount.
- AcceptedCmp1: Response to the 1st campaign (1 for yes, 0 for no).
- AcceptedCmp2: Response to the 2nd campaign (1 for yes, 0 for no).
- AcceptedCmp3: Response to the 3rd campaign (1 for yes, 0 for no).
- AcceptedCmp4: Response to the 4th campaign (1 for yes, 0 for no).
- AcceptedCmp5: Response to the 5th campaign (1 for yes, 0 for no).
- Response: Response to the last campaign (1 for yes, 0 for no).
Shopping Behavior
- NumWebPurchases: Number of purchases made through the company’s website.
- NumCatalogPurchases: Number of purchases made using catalogs.
- NumStorePurchases: Number of purchases made directly in stores.
- NumWebVisitsMonth: Number of visits to the company’s website in the last month.
Let's start coding!¶
Importing necessary libraries¶
In [1]:
# Libraries to help with reading and manipulating data
import pandas as pd
import numpy as np
# libaries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Removes the limit for the number of displayed columns
pd.set_option("display.max_columns", None)
# Sets the limit for the number of displayed rows
pd.set_option("display.max_rows", 200)
# to scale the data using z-score
from sklearn.preprocessing import StandardScaler
# to compute distances
from scipy.spatial.distance import cdist, pdist
# to perform k-means clustering and compute silhouette scores
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
# to visualize the elbow curve and silhouette scores
from yellowbrick.cluster import KElbowVisualizer, SilhouetteVisualizer
# to perform hierarchical clustering, compute cophenetic correlation, and create dendrograms
from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage, cophenet
# to suppress warnings
import warnings
warnings.filterwarnings("ignore")
Loading the data¶
In [2]:
# uncomment and run the following line if using Google Colab
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [94]:
# loading data into a pandas dataframe
data = pd.read_csv("/content/marketing_campaign.csv", sep="\t")
Data Overview¶
Question 1: What are the data types of all the columns?¶
In [95]:
data.info() # returns the data types of all the columns
<class 'pandas.core.frame.DataFrame'> RangeIndex: 2240 entries, 0 to 2239 Data columns (total 29 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 ID 2240 non-null int64 1 Year_Birth 2240 non-null int64 2 Education 2240 non-null object 3 Marital_Status 2240 non-null object 4 Income 2216 non-null float64 5 Kidhome 2240 non-null int64 6 Teenhome 2240 non-null int64 7 Dt_Customer 2240 non-null object 8 Recency 2240 non-null int64 9 MntWines 2240 non-null int64 10 MntFruits 2240 non-null int64 11 MntMeatProducts 2240 non-null int64 12 MntFishProducts 2240 non-null int64 13 MntSweetProducts 2240 non-null int64 14 MntGoldProds 2240 non-null int64 15 NumDealsPurchases 2240 non-null int64 16 NumWebPurchases 2240 non-null int64 17 NumCatalogPurchases 2240 non-null int64 18 NumStorePurchases 2240 non-null int64 19 NumWebVisitsMonth 2240 non-null int64 20 AcceptedCmp3 2240 non-null int64 21 AcceptedCmp4 2240 non-null int64 22 AcceptedCmp5 2240 non-null int64 23 AcceptedCmp1 2240 non-null int64 24 AcceptedCmp2 2240 non-null int64 25 Complain 2240 non-null int64 26 Z_CostContact 2240 non-null int64 27 Z_Revenue 2240 non-null int64 28 Response 2240 non-null int64 dtypes: float64(1), int64(25), object(3) memory usage: 507.6+ KB
Observations:¶
Most are ints, except objects Education, Marital_Status, Dt_Customer, and float for Income
Question 2: Check the statistical summary of the data. What is the average household income?¶
In [96]:
# to do a statistical summary, you do .describe() and you can do T to transpose
data.describe().T
Out[96]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| ID | 2240.0 | 5592.159821 | 3246.662198 | 0.0 | 2828.25 | 5458.5 | 8427.75 | 11191.0 |
| Year_Birth | 2240.0 | 1968.805804 | 11.984069 | 1893.0 | 1959.00 | 1970.0 | 1977.00 | 1996.0 |
| Income | 2216.0 | 52247.251354 | 25173.076661 | 1730.0 | 35303.00 | 51381.5 | 68522.00 | 666666.0 |
| Kidhome | 2240.0 | 0.444196 | 0.538398 | 0.0 | 0.00 | 0.0 | 1.00 | 2.0 |
| Teenhome | 2240.0 | 0.506250 | 0.544538 | 0.0 | 0.00 | 0.0 | 1.00 | 2.0 |
| Recency | 2240.0 | 49.109375 | 28.962453 | 0.0 | 24.00 | 49.0 | 74.00 | 99.0 |
| MntWines | 2240.0 | 303.935714 | 336.597393 | 0.0 | 23.75 | 173.5 | 504.25 | 1493.0 |
| MntFruits | 2240.0 | 26.302232 | 39.773434 | 0.0 | 1.00 | 8.0 | 33.00 | 199.0 |
| MntMeatProducts | 2240.0 | 166.950000 | 225.715373 | 0.0 | 16.00 | 67.0 | 232.00 | 1725.0 |
| MntFishProducts | 2240.0 | 37.525446 | 54.628979 | 0.0 | 3.00 | 12.0 | 50.00 | 259.0 |
| MntSweetProducts | 2240.0 | 27.062946 | 41.280498 | 0.0 | 1.00 | 8.0 | 33.00 | 263.0 |
| MntGoldProds | 2240.0 | 44.021875 | 52.167439 | 0.0 | 9.00 | 24.0 | 56.00 | 362.0 |
| NumDealsPurchases | 2240.0 | 2.325000 | 1.932238 | 0.0 | 1.00 | 2.0 | 3.00 | 15.0 |
| NumWebPurchases | 2240.0 | 4.084821 | 2.778714 | 0.0 | 2.00 | 4.0 | 6.00 | 27.0 |
| NumCatalogPurchases | 2240.0 | 2.662054 | 2.923101 | 0.0 | 0.00 | 2.0 | 4.00 | 28.0 |
| NumStorePurchases | 2240.0 | 5.790179 | 3.250958 | 0.0 | 3.00 | 5.0 | 8.00 | 13.0 |
| NumWebVisitsMonth | 2240.0 | 5.316518 | 2.426645 | 0.0 | 3.00 | 6.0 | 7.00 | 20.0 |
| AcceptedCmp3 | 2240.0 | 0.072768 | 0.259813 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| AcceptedCmp4 | 2240.0 | 0.074554 | 0.262728 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| AcceptedCmp5 | 2240.0 | 0.072768 | 0.259813 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| AcceptedCmp1 | 2240.0 | 0.064286 | 0.245316 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| AcceptedCmp2 | 2240.0 | 0.013393 | 0.114976 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| Complain | 2240.0 | 0.009375 | 0.096391 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
| Z_CostContact | 2240.0 | 3.000000 | 0.000000 | 3.0 | 3.00 | 3.0 | 3.00 | 3.0 |
| Z_Revenue | 2240.0 | 11.000000 | 0.000000 | 11.0 | 11.00 | 11.0 | 11.00 | 11.0 |
| Response | 2240.0 | 0.149107 | 0.356274 | 0.0 | 0.00 | 0.0 | 0.00 | 1.0 |
Observations:¶
Question 3: Are there any missing values in the data? If yes, treat them using an appropriate method¶
In [97]:
#data.isnull().sum()
# there are missing values in the income (theres 24 missing)
data["Income"].describe()
sns.boxplot(data=data, x='Income')
# will use the median to replace missing values
filleddata = data.fillna(data["Income"].median())
filleddata.isnull().sum()
Out[97]:
| 0 | |
|---|---|
| ID | 0 |
| Year_Birth | 0 |
| Education | 0 |
| Marital_Status | 0 |
| Income | 0 |
| Kidhome | 0 |
| Teenhome | 0 |
| Dt_Customer | 0 |
| Recency | 0 |
| MntWines | 0 |
| MntFruits | 0 |
| MntMeatProducts | 0 |
| MntFishProducts | 0 |
| MntSweetProducts | 0 |
| MntGoldProds | 0 |
| NumDealsPurchases | 0 |
| NumWebPurchases | 0 |
| NumCatalogPurchases | 0 |
| NumStorePurchases | 0 |
| NumWebVisitsMonth | 0 |
| AcceptedCmp3 | 0 |
| AcceptedCmp4 | 0 |
| AcceptedCmp5 | 0 |
| AcceptedCmp1 | 0 |
| AcceptedCmp2 | 0 |
| Complain | 0 |
| Z_CostContact | 0 |
| Z_Revenue | 0 |
| Response | 0 |
Observations:¶
There were 24 missing data in the Income section, filled it in with median
Question 4: Are there any duplicates in the data?¶
In [98]:
filleddata = pd.DataFrame(filleddata)
filleddata.duplicated().sum()
Out[98]:
0
Observations:¶
Since filleddata.duplicated().sum() is 0, there are no duplicates
Exploratory Data Analysis¶
Univariate Analysis¶
Question 5: Explore all the variables and provide observations on their distributions. (histograms and boxplots)¶
In [99]:
# list out all of the columns
print(filleddata.columns)
'''
['ID', 'Year_Birth', 'Education', 'Marital_Status', 'Income', 'Kidhome',
'Teenhome', 'Dt_Customer', 'Recency', 'MntWines', 'MntFruits',
'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts',
'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases',
'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth',
'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1',
'AcceptedCmp2', 'Complain', 'Z_CostContact', 'Z_Revenue', 'Response']
'''
Index(['ID', 'Year_Birth', 'Education', 'Marital_Status', 'Income', 'Kidhome',
'Teenhome', 'Dt_Customer', 'Recency', 'MntWines', 'MntFruits',
'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts',
'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases',
'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth',
'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1',
'AcceptedCmp2', 'Complain', 'Z_CostContact', 'Z_Revenue', 'Response'],
dtype='object')
Out[99]:
"\n['ID', 'Year_Birth', 'Education', 'Marital_Status', 'Income', 'Kidhome',\n 'Teenhome', 'Dt_Customer', 'Recency', 'MntWines', 'MntFruits',\n 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts',\n 'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases',\n 'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth',\n 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1',\n 'AcceptedCmp2', 'Complain', 'Z_CostContact', 'Z_Revenue', 'Response']\n"
In [100]:
plt.figure(figsize=(5, 5))
sns.boxplot(data=filleddata['ID'])
# ID seems evenly distributed, which it should be
Out[100]:
<Axes: ylabel='ID'>
In [101]:
plt.figure(figsize=(5, 5))
sns.violinplot(data=filleddata['Year_Birth'])
Out[101]:
<Axes: ylabel='Year_Birth'>
In [102]:
# It seems like around 38% are married, 26% are together, and 21% are single
sns.barplot(filleddata['Marital_Status'].value_counts(normalize=True))
Out[102]:
<Axes: xlabel='Marital_Status', ylabel='proportion'>
In [103]:
# seems to be evenly distributed, with only several outliers
sns.boxplot(data=filleddata, x='Income')
Out[103]:
<Axes: xlabel='Income'>
In [104]:
# seems like around 55% have no kids, and 45% have 1 kit. Only 2% have 2 kids
sns.barplot(filleddata['Kidhome'].value_counts(normalize=True))
Out[104]:
<Axes: xlabel='Kidhome', ylabel='proportion'>
In [105]:
# seems like around 55% have no teens, and 45% have 1 teen. Only 2% have 2 teens
sns.barplot(filleddata['Teenhome'].value_counts(normalize=True))
Out[105]:
<Axes: xlabel='Teenhome', ylabel='proportion'>
In [106]:
# changed Dt_Customer to datetime
filleddata['Dt_Customer'] = pd.to_datetime(filleddata['Dt_Customer'], format='%d-%m-%Y')
filleddata['Dt_Customer'].hist(bins=30)
# seems like the data is quite random and does not have much pattern
Out[106]:
<Axes: >
In [107]:
sns.histplot(data=filleddata, x='Recency')
# looks relatively constant
Out[107]:
<Axes: xlabel='Recency', ylabel='Count'>
In [108]:
segment = pd.DataFrame(filleddata, columns=['MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds'])
from sklearn.preprocessing import MinMaxScaler
# normalizing the data
scaler = MinMaxScaler()
df_normalized = pd.DataFrame(scaler.fit_transform(segment), columns=segment.columns)
for column in df_normalized.columns:
sns.histplot(df_normalized[column], bins=10, kde=False, label=column, alpha=0.1, legend=True)
print(column, filleddata[column].mean())
segment = pd.DataFrame(filleddata, columns=['MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds'])
fig, axes = plt.subplots(nrows=1, ncols=6, figsize=(10, 4))
count = 0
total_avg = 0;
for count, column in enumerate(segment.columns):
sns.boxplot(x=column, data=segment, ax=axes[count])
total_avg += segment[column].mean()
print(column, segment[column].mean())
print('total average', total_avg / len(segment.columns))
# the data seems to skew right for all products
MntWines 303.9357142857143 MntFruits 26.302232142857143 MntMeatProducts 166.95 MntFishProducts 37.52544642857143 MntSweetProducts 27.06294642857143 MntGoldProds 44.021875 MntWines 303.9357142857143 MntFruits 26.302232142857143 MntMeatProducts 166.95 MntFishProducts 37.52544642857143 MntSweetProducts 27.06294642857143 MntGoldProds 44.021875 total average 100.96636904761904
In [109]:
sns.histplot(filleddata['NumDealsPurchases'], kde=True)
# this graph seems to skew right as well
print('mean:', filleddata['NumDealsPurchases'].mean())
print('median:', filleddata['NumDealsPurchases'].median())
# the median would be a better center as the data is skewed
mean: 2.325 median: 2.0
In [110]:
segment = pd.DataFrame(filleddata, columns=['AcceptedCmp1', 'AcceptedCmp2', 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5'])
fig, axes = plt.subplots(nrows=1, ncols=5, figsize=(10, 4))
count = 0
total_avg = 0;
for count, column in enumerate(segment.columns):
sns.countplot(x=column, data=segment, ax=axes[count])
total_avg += segment[column].mean()
print(column, segment[column].mean())
print('total average', total_avg / len(segment.columns))
# looks like on total average, only 6% actually respond to the campaigns.
# however, we see the best performances in campaigns 3, 4, and 5 and
# the worst performances in campaign 2
AcceptedCmp1 0.06428571428571428 AcceptedCmp2 0.013392857142857142 AcceptedCmp3 0.07276785714285715 AcceptedCmp4 0.07455357142857143 AcceptedCmp5 0.07276785714285715 total average 0.05955357142857143
In [111]:
segment = pd.DataFrame(filleddata, columns=['NumWebPurchases', 'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth'])
fig, axes = plt.subplots(nrows=1, ncols=4, figsize=(20, 4))
count = 0
for count, column in enumerate(segment.columns):
sns.countplot(x=column, data=segment, ax=axes[count])
print(column, segment[column].mean())
# for each category, it looks like it is skewed to the right
# for number of web visits a month, it seems skewed left, though
# there seem to be high outlifers for web and catalog purchases
NumWebPurchases 4.084821428571429 NumCatalogPurchases 2.6620535714285714 NumStorePurchases 5.790178571428571 NumWebVisitsMonth 5.316517857142857
Observations:¶
Bivariate Analysis¶
Question 6: Perform multivariate analysis to explore the relationsips between the variables.¶
In [112]:
sns.boxplot(filleddata, x='Education', y='Year_Birth')
Out[112]:
<Axes: xlabel='Education', ylabel='Year_Birth'>
In [113]:
# Adding a Ts_Customer column for data analysis
filleddata['Dt_Customer'] = pd.to_datetime(filleddata['Dt_Customer'], dayfirst=True)
filleddata['Ts_Customer'] = filleddata['Dt_Customer'].view('int64')
In [114]:
# some checking to see how the means of Marital_status correlate with the rest of the data
numericaldata = pd.DataFrame(filleddata, columns=['ID', 'Ts_Customer', 'Year_Birth', 'Kidhome', 'Teenhome', 'Recency', 'MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases', 'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth', 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1', 'AcceptedCmp2', 'Complain', 'Z_CostContact', 'Z_Revenue', 'Response'])
# Perform ANOVA on Marital_Status
import scipy.stats as stats
signif_marital_map = {}
for numcol in numericaldata.columns:
anova_result = stats.f_oneway(
filleddata[filleddata['Marital_Status'] == 'Single'][numcol],
filleddata[filleddata['Marital_Status'] == 'Together'][numcol],
filleddata[filleddata['Marital_Status'] == 'Married'][numcol],
filleddata[filleddata['Marital_Status'] == 'Divorced'][numcol],
filleddata[filleddata['Marital_Status'] == 'Widow'][numcol],
filleddata[filleddata['Marital_Status'] == 'Alone'][numcol],
filleddata[filleddata['Marital_Status'] == 'Absurd'][numcol],
filleddata[filleddata['Marital_Status'] == 'YOLO'][numcol]
)
#print(f'ANOVA F-Statistic: {anova_result.statistic}, p-value: {anova_result.pvalue}')
signif_marital_map.update({numcol : anova_result.pvalue})
filtered_items = {key: value for key, value in signif_marital_map.items() if value < 0.01}
# Printing the result
for key, value in filtered_items.items():
print(f"For column: {key}, pvalue: {value}")
For column: Year_Birth, pvalue: 3.990923485536878e-19 For column: Kidhome, pvalue: 0.006399583359671683 For column: Teenhome, pvalue: 6.0909780334068225e-05 For column: MntFishProducts, pvalue: 0.00022535617627389288 For column: MntGoldProds, pvalue: 0.000774627012989525 For column: Response, pvalue: 1.6526511623348644e-09
In [115]:
# some checking to see how the means of Education correlate with the rest of the data
numericaldata = pd.DataFrame(filleddata, columns=['ID', 'Ts_Customer', 'Year_Birth', 'Kidhome', 'Teenhome', 'Recency', 'MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases', 'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth', 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1', 'AcceptedCmp2', 'Complain', 'Z_CostContact', 'Z_Revenue', 'Response'])
# Perform ANOVA on Marital_Status
import scipy.stats as stats
signif_education_map = {}
for numcol in numericaldata.columns:
anova_result = stats.f_oneway(
filleddata[filleddata['Education'] == 'Graduation'][numcol],
filleddata[filleddata['Education'] == 'PhD'][numcol],
filleddata[filleddata['Education'] == 'Master'][numcol],
filleddata[filleddata['Education'] == 'Basic'][numcol],
filleddata[filleddata['Education'] == '2n Cycle'][numcol]
)
#print(f'ANOVA F-Statistic: {anova_result.statistic}, p-value: {anova_result.pvalue}')
signif_education_map.update({numcol : anova_result.pvalue})
filtered_items = {key: value for key, value in signif_education_map.items() if value < 0.01}
# Printing the result
for key, value in filtered_items.items():
print(f"For column: {key}, pvalue: {value}")
For column: Year_Birth, pvalue: 1.438789567380165e-17 For column: Teenhome, pvalue: 5.6737098842502086e-11 For column: MntWines, pvalue: 7.104155194866556e-24 For column: MntFruits, pvalue: 3.568559444333611e-08 For column: MntMeatProducts, pvalue: 1.8740917742751123e-06 For column: MntFishProducts, pvalue: 4.858405851377749e-10 For column: MntSweetProducts, pvalue: 1.6875726298746767e-09 For column: MntGoldProds, pvalue: 2.5260081126899532e-11 For column: NumWebPurchases, pvalue: 3.888123801969024e-09 For column: NumCatalogPurchases, pvalue: 5.803800019901456e-08 For column: NumStorePurchases, pvalue: 3.44357919348158e-10 For column: NumWebVisitsMonth, pvalue: 6.289746487717863e-05 For column: Response, pvalue: 0.00011794504145284023
In [116]:
plt.figure(figsize=(15,10))
filleddata['Dt_Customer'] = pd.to_datetime(filleddata['Dt_Customer'], dayfirst=True)
filleddata['Ts_Customer'] = filleddata['Dt_Customer'].view('int64')
numericaldata = pd.DataFrame(filleddata, columns=['ID', 'Ts_Customer', 'Year_Birth', 'Kidhome', 'Teenhome', 'Recency', 'MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases', 'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth', 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1', 'AcceptedCmp2', 'Complain', 'Z_CostContact', 'Z_Revenue', 'Response'])
sns.heatmap(numericaldata.corr(), annot=True, fmt='0.2f', square=True)
plt.show()
'''
Key things I notice:
1. People seem less likely to use deals to buy fruits, meat, fish, and sweet products
2. People with more kids seem to get less wines, fruits, meats, fish, sweets, gold,
and make fewer purchases, but significantly more web visits per month
3. Strong correlation between # meat products and catalog purchases. Perhaps they buy meat in bulk?
4. Seems to be if you buy one type of produce, there is a chance you are
probably buying another type as well
5. ID, Recency, Complaints, Marital_Status, Ts_Customer, Z_CostContact, and Z_Revenue
don't seem to have any affects
'''
Out[116]:
"\nKey things I notice:\n1. People seem less likely to use deals to buy fruits, meat, fish, and sweet products\n2. People with more kids seem to get less wines, fruits, meats, fish, sweets, gold,\n and make fewer purchases, but significantly more web visits per month\n3. Strong correlation between # meat products and catalog purchases. Perhaps they buy meat in bulk?\n4. Seems to be if you buy one type of produce, there is a chance you are\n probably buying another type as well\n5. ID, Recency, Complaints, Marital_Status, Ts_Customer, Z_CostContact, and Z_Revenue\n don't seem to have any affects\n"
Observations:¶
NEXT STEPS ARE TO PREPROCESS WITH SCALING
In [117]:
# remove unnecessary columns
strippeddata = filleddata.drop(columns=['ID', 'Education', 'Dt_Customer', 'Recency', 'Complain', 'Marital_Status', 'Z_CostContact', 'Z_Revenue'])
# make sure its datetime format
strippeddata['Ts_Customer'] = pd.to_datetime(strippeddata['Ts_Customer'])
# convert to unix time for int scaling
strippeddata['Ts_Customer'] = strippeddata['Ts_Customer'].astype(int) // 10**9
from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
scaled_data = scaler.fit_transform(strippeddata)
In [118]:
scaled_data = pd.DataFrame(scaled_data, columns=strippeddata.columns)
scaled_data.head()
Out[118]:
| Year_Birth | Income | Kidhome | Teenhome | MntWines | MntFruits | MntMeatProducts | MntFishProducts | MntSweetProducts | MntGoldProds | NumDealsPurchases | NumWebPurchases | NumCatalogPurchases | NumStorePurchases | NumWebVisitsMonth | AcceptedCmp3 | AcceptedCmp4 | AcceptedCmp5 | AcceptedCmp1 | AcceptedCmp2 | Response | Ts_Customer | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.722222 | 0.206299 | 0.0 | 0.0 | 0.960458 | 2.50000 | 2.217593 | 3.404255 | 2.50000 | 1.361702 | 0.5 | 1.00 | 2.00 | -0.2 | 0.25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | -0.882986 |
| 1 | -0.888889 | -0.153812 | 1.0 | 1.0 | -0.338189 | -0.21875 | -0.282407 | -0.212766 | -0.21875 | -0.382979 | 0.0 | -0.75 | -0.25 | -0.6 | -0.25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.696339 |
| 2 | -0.277778 | 0.617737 | 0.0 | 0.0 | 0.525494 | 1.28125 | 0.277778 | 2.106383 | 0.40625 | 0.382979 | -0.5 | 1.00 | 0.00 | 1.0 | -0.50 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.124910 |
| 3 | 0.777778 | -0.755259 | 1.0 | 0.0 | -0.338189 | -0.12500 | -0.217593 | -0.042553 | -0.15625 | -0.404255 | 0.0 | -0.50 | -0.50 | -0.2 | 0.00 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.621680 |
| 4 | 0.611111 | 0.211032 | 1.0 | 0.0 | -0.001041 | 1.09375 | 0.236111 | 0.723404 | 0.59375 | -0.191489 | 1.5 | 0.25 | 0.25 | 0.2 | -0.25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.558507 |
K-means Clustering¶
Question 7 : Select the appropriate number of clusters using the elbow Plot. What do you think is the appropriate number of clusters?¶
In [119]:
'''
Let us now fit k-means algorithm on our scaled data and find out the optimum number of clusters to use.
We will do this in 3 steps:
Initialize a dictionary to store the SSE for each k
Run for a range of Ks and store SSE for each run
Plot the SSE vs K and find the elbow
'''
# Creating copy of the data to store labels from each algorithm
scaled_data_copy = scaled_data.copy(deep=True)
# step 1
WCSS = {}
# step 2 - iterate with multiple K's
for k in range(1, 11):
kmeans = KMeans(n_clusters=k, max_iter=1000)
kmeans.fit(scaled_data)
WCSS[k] = kmeans.inertia_
# step 3 - plot the results
plt.figure(figsize=(10, 5))
plt.plot(list(WCSS.keys()), list(WCSS.values()), 'bx-')
plt.xlabel("Number of clusters")
plt.ylabel("WCSS")
# seems like 2 is the elbow point
Out[119]:
Text(0, 0.5, 'WCSS')
Observations:¶
Question 8 : finalize appropriate number of clusters by checking the silhoutte score as well. Is the answer different from the elbow plot?¶
In [120]:
'''
Let us now fit k-means algorithm on our scaled data and find out the optimum number of clusters to use.
We will do this in 3 steps:
Initialize a dictionary to store the SSE for each k
Run for a range of Ks and store SSE for each run
Plot the SSE vs K and find the elbow
SILHOUTTE SCORES
'''
#Empty dictionary to store the Silhouette score for each value of k
sc = {}
# iterate for a range of Ks and fit the scaled data to the algorithm. Store the Silhouette score for that k
for k in range(2, 10):
kmeans = KMeans(n_clusters=k).fit(scaled_data)
labels = kmeans.predict(scaled_data)
sc[k] = silhouette_score(scaled_data, labels)
#Elbow plot
plt.figure()
plt.plot(list(sc.keys()), list(sc.values()), 'bx-')
plt.xlabel("Number of cluster")
plt.ylabel("Silhouette Score")
plt.show()
# looks like 2 has the highest silhoutte score, its the same score
Observations:¶
Question 9: Do a final fit with the appropriate number of clusters. How much total time does it take for the model to fit the data?¶
In [121]:
import time
n_clusters = 2
kmeans = KMeans(n_clusters=n_clusters, max_iter= 1000, random_state=1)
start_time = time.time()
kmeans.fit(scaled_data)
end_time = time.time()
duration = end_time - start_time
# Adding predicted labels to the original data and scaled data
scaled_data_copy['Labels'] = kmeans.predict(scaled_data)
data['Labels'] = kmeans.predict(scaled_data)
print(f"K-means fitting took {duration:.4f} seconds for {n_clusters} clusters.")
K-means fitting took 0.0129 seconds for 2 clusters.
In [122]:
# See the number of observations in each cluster
data.Labels.value_counts()
Out[122]:
| count | |
|---|---|
| Labels | |
| 1 | 1506 |
| 0 | 734 |
Observations:¶
Hierarchical Clustering¶
Question 10: Calculate the cophnetic correlation for every combination of distance metrics and linkage. Which combination has the highest cophnetic correlation?¶
In [123]:
# Going to apply PCA to optimize data set
from sklearn.decomposition import PCA
X_reduced = PCA(n_components=2).fit_transform(scaled_data.copy())
hc_df = X_reduced
hc_df1 = X_reduced.copy()
# To perform hierarchical clustering, compute cophenetic correlation, and create dendrograms
from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage, cophenet
from scipy.spatial.distance import pdist
# List of distance metrics
distance_metrics = ["euclidean", "chebyshev", "cityblock"] # not using mahalanobis
# List of linkage methods
linkage_methods = ["single", "complete", "average", "weighted"]
high_cophenet_corr = 0
high_dm_lm = [0, 0]
for dm in distance_metrics:
for lm in linkage_methods:
Z = linkage(hc_df1, metric = dm, method = lm)
c, coph_dists = cophenet(Z, pdist(hc_df))
print(
"Cophenetic correlation for {} distance and {} linkage is {}.".format(
dm.capitalize(), lm, c
)
)
if high_cophenet_corr < c:
high_cophenet_corr = c
high_dm_lm[0] = dm
high_dm_lm[1] = lm
# Printing the combination of distance metric and linkage method with the highest cophenetic correlation
print('*'*100)
print(
"Highest cophenetic correlation is {}, which is obtained with {} distance and {} linkage.".format(
high_cophenet_corr, high_dm_lm[0].capitalize(), high_dm_lm[1]
)
)
Cophenetic correlation for Euclidean distance and single linkage is 0.5337475287779134. Cophenetic correlation for Euclidean distance and complete linkage is 0.7702570176689502. Cophenetic correlation for Euclidean distance and average linkage is 0.7926813567958819. Cophenetic correlation for Euclidean distance and weighted linkage is 0.7763443464553523. Cophenetic correlation for Chebyshev distance and single linkage is 0.5517141377053149. Cophenetic correlation for Chebyshev distance and complete linkage is 0.789277750086223. Cophenetic correlation for Chebyshev distance and average linkage is 0.8167594468206901. Cophenetic correlation for Chebyshev distance and weighted linkage is 0.7666169308294313. Cophenetic correlation for Cityblock distance and single linkage is 0.5324960580459047. Cophenetic correlation for Cityblock distance and complete linkage is 0.7813659410394007. Cophenetic correlation for Cityblock distance and average linkage is 0.7998716823947444. Cophenetic correlation for Cityblock distance and weighted linkage is 0.690934999453801. **************************************************************************************************** Highest cophenetic correlation is 0.8167594468206901, which is obtained with Chebyshev distance and average linkage.
In [124]:
high_cophenet_corr = 0
high_dm_lm = [0, 0]
for lm in linkage_methods:
Z = linkage(hc_df1, metric = "euclidean", method = lm)
c, coph_dists = cophenet(Z, pdist(hc_df))
print("Cophenetic correlation for {} linkage is {}.".format(lm, c))
if high_cophenet_corr < c:
high_cophenet_corr = c
high_dm_lm[0] = "euclidean"
high_dm_lm[1] = lm
# Printing the combination of distance metric and linkage method with the highest cophenetic correlation
print('*'*100)
print(
"Highest cophenetic correlation is {}, which is obtained with {} linkage.".format(
high_cophenet_corr, high_dm_lm[1]
)
)
Cophenetic correlation for single linkage is 0.5337475287779134. Cophenetic correlation for complete linkage is 0.7702570176689502. Cophenetic correlation for average linkage is 0.7926813567958819. Cophenetic correlation for weighted linkage is 0.7763443464553523. **************************************************************************************************** Highest cophenetic correlation is 0.7926813567958819, which is obtained with average linkage.
Observations:¶
Question 11: plot the dendogram for every linkage method with "Euclidean" distance only. What should be the appropriate linkage according to the plot?¶
In [125]:
# View the dendrograms
# Lists to save results of cophenetic correlation calculation
compare_cols = ["Linkage", "Cophenetic Coefficient"]
compare = []
# To create a subplot image
fig, axs = plt.subplots(len(linkage_methods), 1, figsize = (15, 30))
# We will enumerate through the list of linkage methods above
# For each linkage method, we will plot the dendrogram and calculate the cophenetic correlation
for i, method in enumerate(linkage_methods):
Z = linkage(hc_df1, metric = "euclidean", method = method)
dendrogram(Z, ax = axs[i])
axs[i].set_title(f"Dendrogram ({method.capitalize()} Linkage)")
coph_corr, coph_dist = cophenet(Z, pdist(hc_df))
axs[i].annotate(
f"Cophenetic\nCorrelation\n{coph_corr:0.2f}",
(0.80, 0.80),
xycoords="axes fraction",
)
compare.append([method, coph_corr])
In [126]:
# The Average Linkage (0.79) and Weighted Linkage (0.78) seemed to have the best cophenetic correlation
Observations:¶
Question 12: Check the silhoutte score for the hierchial clustering. What should be the appropriate number of clusters according to this plot?¶
In [127]:
from scipy.cluster.hierarchy import linkage, fcluster
# using df_hc1 as dataset
Z = linkage(hc_df1, method='ward') # Use the best method from previous analysis
# Range of number of clusters to evaluate
range_n_clusters = range(2, 10) # Adjust range based on your dataset
silhouette_avg_scores = []
for n_clusters in range_n_clusters:
# Generate cluster labels
cluster_labels = fcluster(Z, n_clusters, criterion='maxclust')
# Calculate the silhouette score
if len(set(cluster_labels)) > 1: # silhouette_score requires at least 2 clusters
silhouette_avg = silhouette_score(hc_df1, cluster_labels)
silhouette_avg_scores.append(silhouette_avg)
print(f"For n_clusters = {n_clusters}, the silhouette score is {silhouette_avg:.4f}")
else:
silhouette_avg_scores.append(-1) # handle cases with only one cluster
# Plotting the silhouette scores
plt.figure(figsize=(10, 6))
plt.plot(range_n_clusters, silhouette_avg_scores, marker='o')
plt.xticks(range_n_clusters)
plt.xlabel('Number of clusters')
plt.ylabel('Average silhouette score')
plt.title('Silhouette scores for hierarchical clustering')
plt.grid(True)
plt.show()
# Determine the optimal number of clusters based on the silhouette score
optimal_clusters = range_n_clusters[np.argmax(silhouette_avg_scores)]
print(f"The optimal number of clusters according to the silhouette score is: {optimal_clusters}")
For n_clusters = 2, the silhouette score is 0.4951 For n_clusters = 3, the silhouette score is 0.4323 For n_clusters = 4, the silhouette score is 0.3789 For n_clusters = 5, the silhouette score is 0.3800 For n_clusters = 6, the silhouette score is 0.3727 For n_clusters = 7, the silhouette score is 0.3646 For n_clusters = 8, the silhouette score is 0.3698 For n_clusters = 9, the silhouette score is 0.3391
The optimal number of clusters according to the silhouette score is: 2
Observations:¶
Question 13: Fit the Hierarchial clustering model with the appropriate parameters finalized above. How much time does it take to fit the model?¶
In [128]:
optimal_linkage = high_dm_lm[1]
optimal_distance = high_dm_lm[0]
# Assuming the optimal number of clusters found via silhouette scores or prior analysis
optimal_clusters = 2 # We found from the silhoutte model score that 2 is best
# Fit the hierarchical clustering model
model = AgglomerativeClustering(n_clusters=optimal_clusters, linkage=optimal_linkage)
start_time = time.time()
model.fit(hc_df1)
end_time = time.time()
# Obtain the labels
cluster_labels = model.labels_
# You may want to print the cluster labels or use them in further analysis
print(f"Cluster labels assigned: {np.unique(cluster_labels)}")
duration = end_time - start_time
# Adding predicted labels to the original data and scaled data
scaled_data_copy['Labels'] = kmeans.predict(scaled_data)
data['Labels'] = kmeans.predict(scaled_data)
print(f"K-means fitting took {duration:.4f} seconds for {optimal_clusters} clusters.")
Cluster labels assigned: [0 1] K-means fitting took 0.1609 seconds for 2 clusters.
Observations:¶
Cluster Profiling and Comparison¶
K-Means Clustering vs Hierarchical Clustering Comparison¶
Question 14: Perform and compare Cluster profiling on both algorithms using boxplots. Based on the all the observaions Which one of them provides better clustering?¶
In [129]:
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.cluster import KMeans, AgglomerativeClustering
# Assuming scaled_data is a DataFrame that includes your features to cluster on
# Ensure that scaled_data_copy and data are separate DataFrames if required
# Profiling for K-Means
kmeans_labels = kmeans.predict(scaled_data)
filleddata['KMeans_Labels'] = kmeans_labels
# Profiling for Agglomerative Clustering
agglomerative_labels = model.labels_
filleddata['Agglomerative_Labels'] = agglomerative_labels
# List your feature columns for profiling
feature_columns = scaled_data.columns
# Function to display cluster profile data
def plot_cluster_alternating(filleddata, feature_columns, title):
num_features = len(feature_columns)
num_rows = (num_features + 1) // 2 # Two features per row for alternating display
fig, axes = plt.subplots(num_rows, 2, figsize=(14, num_rows * 5))
for i, feature in enumerate(feature_columns):
row = i // 2
col = i % 2
sns.boxplot(ax=axes[row, col], x='Cluster', y=feature, hue='Cluster Method', data=filleddata)
axes[row, col].set_title(f'{title}: {feature}')
axes[row, col].set_xlabel('Cluster')
axes[row, col].set_ylabel('Value')
# Remove empty subplots if the features count is odd
if num_features % 2 != 0:
fig.delaxes(axes[-1, -1])
fig.tight_layout(pad=3.0)
plt.show()
filleddata_long_kmeans = filleddata.copy()
filleddata_long_kmeans['Cluster Method'] = 'KMeans'
filleddata_long_kmeans = filleddata_long_kmeans.rename(columns={'KMeans_Labels': 'Cluster'})
filleddata_long_agglo = filleddata.copy()
filleddata_long_agglo['Cluster Method'] = 'Agglomerative'
filleddata_long_agglo = filleddata_long_agglo.rename(columns={'Agglomerative_Labels': 'Cluster'})
# Concatenate the long-form data
filleddata_long_form = pd.concat([filleddata_long_kmeans, filleddata_long_agglo], ignore_index=True)
# Call the function to plot
plot_cluster_alternating(filleddata_long_form, feature_columns, 'Comparison of Clustering Methods')
# Function to compute covariance matrices for clusters
def compute_covariance_summary(filleddata, feature_columns, cluster_column, cluster_label):
cluster_data = filleddata[filleddata[cluster_column] == cluster_label][feature_columns]
cov_matrix = np.cov(cluster_data, rowvar=False)
return np.trace(cov_matrix) # Sum of variances (trace of the covariance matrix)
# Display covariance matrices
kmeans_cov_summary = sum(
compute_covariance_summary(filleddata, feature_columns, 'KMeans_Labels', label)
for label in np.unique(filleddata['KMeans_Labels'])
)
print("K-Means Covariance Summary:", kmeans_cov_summary)
print('--------------------------------------')
agglo_cov_summary = sum(
compute_covariance_summary(filleddata, feature_columns, 'Agglomerative_Labels', label)
for label in np.unique(filleddata['Agglomerative_Labels'])
)
print("Hierarchical Covariance Summary:", agglo_cov_summary)
print('--------------------------------------')
K-Means Covariance Summary: 6.061788512044852e+32 -------------------------------------- Hierarchical Covariance Summary: 6.119508026045693e+32 --------------------------------------
Observations:¶
Both work best with 2 clusters, however, there is a difference in the distribution of each cluster with the algorithms. The two clusters in the Agglomerative algorithm are distributed a little more similarly whereas using Kmeans was a little less evenly distributed.
Question 15: Perform Cluster profiling on the data with the appropriate algorithm determined above using a barplot. What observations can be derived for each cluster from this plot?¶
In [130]:
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.cluster import KMeans, AgglomerativeClustering
# Assuming scaled_data is a DataFrame that includes your features to cluster on
# Ensure that scaled_data_copy and data are separate DataFrames if required
# Profiling for K-Means
kmeans_labels = kmeans.predict(scaled_data)
filleddata['KMeans_Labels'] = kmeans_labels
# Profiling for Agglomerative Clustering
agglomerative_labels = model.labels_
filleddata['Agglomerative_Labels'] = agglomerative_labels
# List your feature columns for profiling
feature_columns = scaled_data.columns
def plot_cluster_alternating(filleddata, feature_columns, title):
num_features = len(feature_columns)
num_rows = (num_features + 1) // 2 # Two features per row for alternating display
fig, axes = plt.subplots(num_rows, 2, figsize=(14, num_rows * 5))
for i, feature in enumerate(feature_columns):
row = i // 2
col = i % 2
sns.barplot(ax=axes[row, col], x='Cluster', y=feature, hue='Cluster Method', data=filleddata)
axes[row, col].set_title(f'{title}: {feature}')
axes[row, col].set_xlabel('Cluster')
axes[row, col].set_ylabel('Value')
# Remove empty subplots if the features count is odd
if num_features % 2 != 0:
fig.delaxes(axes[-1, -1])
fig.tight_layout(pad=3.0)
plt.show()
filleddata_long_kmeans = filleddata.copy()
filleddata_long_kmeans['Cluster Method'] = 'KMeans'
filleddata_long_kmeans = filleddata_long_kmeans.rename(columns={'KMeans_Labels': 'Cluster'})
filleddata_long_agglo = filleddata.copy()
filleddata_long_agglo['Cluster Method'] = 'Agglomerative'
filleddata_long_agglo = filleddata_long_agglo.rename(columns={'Agglomerative_Labels': 'Cluster'})
# Concatenate the long-form data
filleddata_long_form = pd.concat([filleddata_long_kmeans, filleddata_long_agglo], ignore_index=True)
# Call the function to plot
plot_cluster_alternating(filleddata_long_form, feature_columns, 'Comparison of Clustering Methods')
# Function to compute covariance matrices for clusters
def compute_covariance_summary(filleddata, feature_columns, cluster_column, cluster_label):
cluster_data = filleddata[filleddata[cluster_column] == cluster_label][feature_columns]
cov_matrix = np.cov(cluster_data, rowvar=False)
return np.trace(cov_matrix) # Sum of variances (trace of the covariance matrix)
# Display covariance matrices
kmeans_cov_summary = sum(
compute_covariance_summary(filleddata, feature_columns, 'KMeans_Labels', label)
for label in np.unique(filleddata['KMeans_Labels'])
)
print("K-Means Covariance Summary:", kmeans_cov_summary)
print('--------------------------------------')
agglo_cov_summary = sum(
compute_covariance_summary(filleddata, feature_columns, 'Agglomerative_Labels', label)
for label in np.unique(filleddata['Agglomerative_Labels'])
)
print("Hierarchical Covariance Summary:", agglo_cov_summary)
print('--------------------------------------')
K-Means Covariance Summary: 6.061788512044852e+32 -------------------------------------- Hierarchical Covariance Summary: 6.119508026045693e+32 --------------------------------------
Observations:¶
Just like how we saw with the boxplots, both work best with 2 clusters, however, there is a difference in the distribution of each cluster with the algorithms. The two clusters in the Agglomerative algorithm are distributed a little more similarly whereas using Kmeans was a little less evenly distributed.
Business Application¶
Cluster 1:
- ~40k income
- has 0.6 kids home
- has 0.6 teens home
- does not purchase wine, fruits, meat, fish, sweet, gold much
- uses significantly more deals to purchase
- less web purchases
- more catalog purchases
- less store purchases
- more web visits per month
- liked the more recent campaigns
- less response in general
Cluster 2:
- ~75k income
- basically no kids, probably no teens home
- purchases wine, fruits, meat, fish, sweet, gold much more
- does not use deals as much
- makes more web, store, and catalog purchases
- less web visits a month
- generally responsive to all campaigns
To better engage both customer clusters, the business should take a targeted approach rather than a one-size-fits-all strategy.
For Cluster 1, which consists of lower-income shoppers who are more deal-oriented and prefer catalog purchases, the focus should be on maximizing value and convenience. These customers are always on the lookout for discounts, so offering personalized deal alerts through email, SMS, or an app could help increase conversions. Since they make fewer store and web purchases but favor catalogs, enhancing catalog-exclusive promotions or loyalty-based mailer discounts could be a game-changer. They also visit the website often but don’t buy as much, meaning a well-placed flash sale, time-sensitive discount, or a gamified shopping experience could encourage them to complete a purchase. Lastly, they’ve responded well to recent campaigns but aren’t highly engaged overall, so shorter, more frequent promotions with immediate rewards would likely work better than long-term engagement strategies.
On the other hand, Cluster 2 has a higher income, makes frequent purchases across all channels, and is much less focused on discounts. They tend to buy premium items like wine, fresh meat, seafood, sweets, and even gold, meaning the best approach is elevating the premium shopping experience rather than offering discounts. Instead of generic promotions, the business should highlight exclusive memberships, early access to high-end products, and personalized product recommendations to keep them engaged. They shop both online and in-store but visit the website less frequently, so streamlining the online shopping experience with one-click reordering, priority shipping, or subscription boxes could help increase digital engagement. Since they respond well to all campaigns, the best strategy here would be experience-driven marketing—like wine-tasting events, chef-curated meal pairings, or behind-the-scenes product storytelling that makes them feel like part of something exclusive.
Ultimately, the business should focus on segmented marketing rather than blanket promotions. For Cluster 1, the goal is to create enticing, value-driven deals that match their buying behavior, while Cluster 2 would benefit from a more premium, personalized experience that reinforces their purchasing habits. Optimizing the website, refining the catalog strategy, and structuring loyalty programs differently for each group will ensure both segments feel catered to and increase overall sales.
PLEASE NOTE: This is not my dataset, nor my question template. Thank you for reading!