🏡 Data-Driven Real Estate Appraisal Using Machine Learning Techniques

• Advisor: Dr. Ashraf Alsaeed
• Student Name: Fahad Abubaker Bahashwan
• Institution: MidOcean University
• Date: 13, August 2025

What we'll do:

1. Load and explore data
2. Create geographic features
3. Train different ML models
4. Compare which model works best

📚 Load Required Libraries

Import all the essential Python libraries needed for data processing, analysis, visualization, and modeling.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from geopy.distance import geodesic
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import Ridge, Lasso, LinearRegression
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score 
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import xgboost as xgb

⚙️ Initialize Default Values

Define default configuration settings, constants, or initial parameters to be used throughout the notebook.

city_center_coords = {
    'Riyadh': (24.702157462388044, 46.74276616894144),
    'Jeddah': (21.55406527981118, 39.18189665824677),
    'Mecca': (21.42330879427541, 39.82553438746125),
    'Medina': (24.45808931439141, 39.62228459678812),
    'Dammam': (26.40491223507235, 50.082472821265576),
    'Khobar': (26.271817530349022, 50.204861924141625),
    'Qatif': (26.5656, 50.0089),
    'Al-Ahsa': (26.567036126182284, 50.005786609509734),
    'Jubail': (27.003708651215092, 49.629486676571645),
    'Khamis Mushait': (18.30250384919938, 42.74398498622415),
    'Buraidah': (26.356200460399695, 43.968039364588286),
    'Hafar Al-Batin': (28.41245379479697, 45.96700913116033),
    'Diriyah': (24.75019708336476, 46.53413641074703),
    'Huraymila': (25.112716354278646, 46.09974748404961),
    'Nairyah': (27.473500934006946, 48.48032843080484),
    'Jumum': (21.622696847405454, 39.69755012299724)
}

# Mapping Arabic province names to English
province_map = {
    'الرياض': 'Riyadh',
    'جدة': 'Jeddah',
    'مكة المكرمة': 'Mecca',
    'المدينة المنورة': 'Medina',
    'الدمام': 'Dammam',
    'الخبر': 'Khobar',
    'القطيف': 'Qatif',
    'الاحساء': 'Al-Ahsa',
    'الجبيل': 'Jubail',
    'خميس مشيط': 'Khamis Mushait',
    'بريدة': 'Buraidah',
    'حفر الباطن': 'Hafar Al-Batin',
    'الدرعية': 'Diriyah',
    'حـريملاء': 'Huraymila',
    'النعيرية': 'Nairyah',
    'الجموم': 'Jumum'
}




property_type = {
    'أرض سكني': 'Residential Land',
    'فلل': 'Villas',
    'شقق': 'Apartments',
    'مرافق': 'Facilities',
    'زراعي': 'Agricultural',
    'أرض تجاري': 'Commercial Land',
    'مصانع ومستودعات': 'Factories & Warehouses',
    'أرض تجاري سكني': 'Mixed-use Land',
    'اخرى': 'Other',
    'مبنى تجاري سكني': 'Mixed-use Building',
    'مبنى سكني/تجاري': 'Mixed-use Building',
    'استراحات': 'Rest Houses',
    'أرض فضاء': 'Vacant Land',
    'مبنى تجاري': 'Commercial Building',
    'شقة': 'Apartments',
    'مرفق صناعي': 'Industrial Facility',
    'استراحة': 'Rest House',
    'مبنى سكني': 'Residential Building',
    'شاليه': 'Chalet'
}

pd.set_option('display.max_columns', None)

📂 Load Original Dataset and Create a Working Copy

Import the original dataset and create a duplicate to preserve the raw data for reference and ensure safe preprocessing.
The dataset has been successfully loaded and contains

orginal_df = pd.read_excel("AllTransa_10082025_5.xlsx")

df = orginal_df.copy()

df.shape

(66005, 55)

# Calculate the ratio / priceOfMeter_median
df['diff_ratio'] = (df['priceOfMeter'] - df['avaragePriceOfMeter']) / df['priceOfMeter'] 

df = df[(df['diff_ratio'] >= -0.20) & (df['diff_ratio'] <= 0.20)]

df.shape

(38307, 56)

# Define intervals (edges) including "larger than 5000"
bins = [0, 500, 1300, 2000, 2700, 3800, 4500, 5500, 6800, 8500, 9500,12000, float('inf')]

# Create readable labels for each interval
labels = [f"{bins[i]} - {bins[i+1]}" if bins[i+1] != float('inf') else f">{bins[i]}" 
          for i in range(len(bins)-1)]

# Use pd.cut with manual bins
df['Price_interval'] = pd.cut(df['priceOfMeter'], bins=bins, labels=labels, right=True)

# Count how many fall in each interval
interval_counts = df['Price_interval'].value_counts().sort_index() 

interval_counts.head(100)

Price_interval
0 - 500         1368
500 - 1300      7430
1300 - 2000     6495
2000 - 2700     4662
2700 - 3800     6553
3800 - 4500     2950
4500 - 5500     2173
5500 - 6800     1859
6800 - 8500     1928
8500 - 9500      884
9500 - 12000    1355
>12000           650
Name: count, dtype: int64

🛠️ Clean Column Names and Standardize Formats

Refactor column names for consistency, convert Arabic labels to English, and ensure proper data formatting for analysis.

# Convert to datetime
df['transactionDate'] = pd.to_datetime(df['transactionDate'], errors='coerce')
# Add month feature (1 to 12)
df['month'] = df['transactionDate'].dt.month
# Add week of year (1 to 52)
df['week'] = df['transactionDate'].dt.isocalendar().week

df['transactionDate'] = df['transactionDate'].dt.strftime('%Y-%m-%d')

# convert from Arabic to English names
df['metricsType'] = df['metricsType'].map(property_type)
df['provinceName'] = df['provinceName'].map(province_map)
# if average -1 then change to be priceofmeter
df.loc[df['avaragePriceOfMeter'].eq(-1), 'avaragePriceOfMeter'] = df['priceOfMeter']

df['provinceName'].value_counts()

provinceName
Riyadh            23302
Jeddah             4019
Dammam             2499
Mecca              2074
Khobar             1595
Buraidah           1336
Al-Ahsa            1244
Medina              577
Huraymila           569
Jubail              377
Qatif               302
Khamis Mushait      165
Jumum                91
Hafar Al-Batin       86
Diriyah              48
Nairyah              23
Name: count, dtype: int64

🧹 Remove Unnecessary Columns and Reorder Features

Drop irrelevant or redundant columns and rearrange the remaining features into a logical, consistent order.

# Define the columns to drop as a variable
columns_to_drop = [
    'main_key', '_priceOfMeter', 'transactionPrice',  'type', 'subdivisionId', 'polygonData', 'noOfProperties', 'zoningId',
    'parcelId', 'parcelObjectId',  'blockNo', 
    'parcelImageURL', 'projectName', 'landUsageGroup', 'sellingType', 'landUseaDetailed',
    'centroid', 'propertyType', 'totalArea', 'details', 'landUseGroup',
    'orignalTransactionNum',

] #'area', 
#'isLowValueTransaction','transactionSource',
#,neighborhood', 'neighborhoodId', 'region', 'regionId', 'provinceId'
# ,'subdivisionNo' , 'parcelNo',
# Drop the columns using the variable
df = df.drop(columns=columns_to_drop)
 

df['metricsType'].value_counts()

metricsType
Residential Land          17305
Apartments                10309
Mixed-use Land             3487
Villas                     2584
Facilities                 2007
Agricultural                887
Factories & Warehouses      804
Commercial Land             647
Mixed-use Building          232
Commercial Building          23
Rest Houses                  22
Name: count, dtype: int64

df.shape

(38307, 37)

values_to_remove = ['Facilities','Agricultural','Factories & Warehouses','Other','Vacant Land','Commercial Building',
                   'Rest Houses','Residential Building','Rest House','Industrial Facility','Chalet','Mixed-use Building'
                  
                   ]

df = df[~df['metricsType'].isin(values_to_remove)]

df['metricsType'].value_counts()

metricsType
Residential Land    17305
Apartments          10309
Mixed-use Land       3487
Villas               2584
Commercial Land       647
Name: count, dtype: int64

df.shape

(34332, 37)

df = df[df['isLowValueTransaction'] != True]   ## Remove unreal values
df.shape

(34331, 37)

🧼 Clean Price Data: Remove Zero Values and Outliers

1. Remove zero values from the priceOfMeter column to eliminate invalid entries.
2. Apply IQR filtering (Interquartile Range) to remove extreme outliers and retain data within a statistically reasonable range.
   • Outliers are defined as values outside Q1 - 1.5 * IQR and Q3 + 1.5 * IQR.

# 1. Remove zero prices
df2 = df[df['priceOfMeter'] > 0]

# 2. Remove outliers using IQR (Interquartile Range)
Q1 = df2['priceOfMeter'].quantile(0.25)
Q3 = df2['priceOfMeter'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Filter rows within bounds
df2 = df2[(df2['priceOfMeter'] >= lower_bound) & (df2['priceOfMeter'] <= upper_bound)]

df2.describe()
df2.shape

(31651, 37)

# Plot the boxplot for the column 'priceofmeter'
plt.figure(figsize=(8, 5))
sns.boxplot(data=df2, x='priceOfMeter')

plt.title('Boxplot of Price per Meter')
plt.xlabel('Price per Meter')
plt.grid(True)
plt.show()

➕ Feature Engineering: Add New Features

Create additional meaningful features to enhance model performance and capture deeper insights from the data.

# Calculate distance to city center using haversine Formula - https://en.wikipedia.org/wiki/Haversine_formula

def haversine_np_m(lat1, lon1, lat2, lon2):
    R = 6_371_008.8  # meters
    lat1, lon1, lat2, lon2 = map(np.radians, [lat1, lon1, lat2, lon2])
    dlat = lat2 - lat1
    dlon = lon2 - lon1
    a = np.sin(dlat/2.0)**2 + np.cos(lat1)*np.cos(lat2)*np.sin(dlon/2.0)**2
    c = 2 * np.arcsin(np.sqrt(a))
    return R * c

# get (center_lat, center_lon) for each row's city
centers = df['provinceName'].map(city_center_coords).apply(pd.Series)
centers.columns = ['center_lat', 'center_lon']

# straight-line distance to city center in *meters*
df2['dist_to_city_center_m'] = haversine_np_m(
    df2['centroidY'], df['centroidX'],
    centers['center_lat'], centers['center_lon']
)

# Calculate Nearby Schools Count ( 1KM  )
df_schools = pd.read_excel("schools.xlsx")
# Define haversine (vectorized)  - https://en.wikipedia.org/wiki/Haversine_formula
def haversine_np(lat1, lon1, lat2, lon2):
    R = 6371  # Earth radius in KM
    lat1, lon1, lat2, lon2 = map(np.radians, [lat1, lon1, lat2, lon2])
    dlat = lat2 - lat1
    dlon = lon2 - lon1
    a = np.sin(dlat/2.0)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2.0)**2
    c = 2 * np.arcsin(np.sqrt(a))
    return R * c

# Function to count schools within 3km for a target point
def count_schools_near_target(row, schools_df, radius_km=1):
    distances = haversine_np(row['centroidY'], row['centroidX'],
                             schools_df['latitude'].values, schools_df['longitude'].values)
    return int(np.sum(distances <= radius_km))

# Apply to each row in 
df2['schools_within_1km'] = df2.apply(
    lambda row: count_schools_near_target(row, df_schools), axis=1
)

# Convert `metricsType` into one-hot encoded columns and standardize column names for consistency.

df2 = pd.get_dummies(df2, columns=['metricsType'], prefix='', prefix_sep='')

# Clean column names
df2.columns = (
    df2.columns
    .str.strip()                       # Remove leading/trailing whitespace
    .str.replace(' ', '_', regex=False)  # Replace spaces with _
    .str.replace('-', '_', regex=False)  # Replace dashes with _
    .str.lower()                        # Convert to lowercase
)

df2.dropna(inplace=True)  # if exist

df2.shape

(31327, 43)

df2.duplicated().sum()

9

df2.drop_duplicates(inplace=True)

# ✅ Define one-hot encoded type columns
type_cols = ['apartments', 'commercial_land', 'residential_land',  'mixed_use_land', 'villas'] #'mixed_use_building',
 
# ✅ Label the original data for plotting
df2_long = []
for col in type_cols:
    temp = df2[df2[col] == 1].copy()
    temp['type'] = col
    temp['source'] = 'Before'
    df2_long.append(temp)
df2_before = pd.concat(df2_long)

# ✅ Clean the data using IQR per type
cleaned_dfs = []
for col in type_cols:
    type_df = df2[df2[col] == 1]
    Q1 = type_df['priceofmeter'].quantile(0.25)
    Q3 = type_df['priceofmeter'].quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR
    filtered = type_df[(type_df['priceofmeter'] >= lower) & (type_df['priceofmeter'] <= upper)].copy()
    filtered['type'] = col
    filtered['source'] = 'After'
    cleaned_dfs.append(filtered)
df2_after = pd.concat(cleaned_dfs)

# ✅ Combine before and after for plotting
df2_plot = pd.concat([df2_before, df2_after])

# ✅ Create the boxplot
plt.figure(figsize=(14, 6))
sns.boxplot(data=df2_plot, x='type', y='priceofmeter', hue='source')
plt.title('Price of Meter: Before vs After Outlier Removal (per Property Type)')
plt.xlabel('Property Type')
plt.ylabel('Price per Meter')
plt.xticks(rotation=15)
plt.grid(True)
plt.tight_layout()
plt.show()

#df2_before.shape
df2_after.shape

(29747, 45)

🔥 Correlation Heatmap of Numerical Features

Visualize the pairwise correlations between all numerical features to identify strong linear relationships and potential multicollinearity.

 
selected_cols = ['priceofmeter', 'centroidx', 'centroidy', 'dist_to_city_center_m','schools_within_1km',
               'total_road_length_1km',	'count_motorway_1km',	'count_trunk_1km',	'count_primary_1km',	'count_secondary_1km',
                 'count_tertiary_1km',	'count_residential_1km',	'road_type_diversity_1km',	'dist_to_motorway',	'dist_to_trunk',
                 'dist_to_primary',	'dist_to_secondary',	'dist_to_tertiary',	'dist_to_residential'] 
    
numeric_df = df2_after[selected_cols]

plt.figure(figsize=(10, 8))
sns.heatmap(numeric_df.corr(), annot=True, cmap='coolwarm')
plt.title("Correlation Heatmap (Selected Columns)")
plt.show()

🏗️ Feature Selection and Model Training Setup

Select key features to train the model and define the target variable priceofmeter.

features = ['centroidy', 'centroidx','schools_within_1km', 'dist_to_city_center_m', 'apartments',
            'commercial_land', 'residential_land',  'mixed_use_land', 'villas',
            'total_road_length_1km',	'count_motorway_1km',	'count_trunk_1km',	'count_primary_1km',	'count_secondary_1km',
                 'count_tertiary_1km',	'count_residential_1km',	'road_type_diversity_1km',	'dist_to_motorway',	'dist_to_trunk',
                 'dist_to_primary',	'dist_to_secondary',	'dist_to_tertiary',	'dist_to_residential' ] #,
target = 'priceofmeter'
X = df2_after[features]
y = df2_after[target]

🤖 Train Multiple Models and Evaluate Performance

Split the data into training and testing sets, then apply preprocessing using pipelines:

• Numerical features: Imputation and standard scaling
• Categorical features: Passed through as-is

Train and evaluate multiple regression models, including:
LinearRegression, Ridge, Lasso, RandomForest, GradientBoosting, KNN, DecisionTree, and XGBoost.
Performance is assessed using R² and RMSE on both training and test sets.

 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) #42

 
numeric_features = ['centroidy', 'centroidx','schools_within_1km', 'dist_to_city_center_m',
                    'total_road_length_1km',	'count_motorway_1km',	'count_trunk_1km',	'count_primary_1km',	'count_secondary_1km',
                 'count_tertiary_1km',	'count_residential_1km',	'road_type_diversity_1km',	'dist_to_motorway',	'dist_to_trunk',
                 'dist_to_primary',	'dist_to_secondary',	'dist_to_tertiary',	'dist_to_residential'  
                    ] 

categorical_features = ['apartments', 'commercial_land', 'residential_land',   'mixed_use_land', 'villas']
 
numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy="mean")),
    ('scaler', StandardScaler())
])

 
preprocessor = ColumnTransformer([
    ('num', numeric_transformer, numeric_features),
    ('cat', 'passthrough', categorical_features)
])

 
models = {
    'LinearRegression': LinearRegression(),
    'Ridge': Ridge(),
    'Lasso': Lasso(),
    'RandomForest': RandomForestRegressor(n_estimators=50, random_state=42),
    'GradientBoosting': GradientBoostingRegressor(n_estimators=50, random_state=42),
    'KNN': KNeighborsRegressor(n_neighbors=5),
    'DecisionTree': DecisionTreeRegressor(random_state=42),
    'XGBoost': xgb.XGBRegressor(n_estimators=50, random_state=42)
} 

📊 Model Performance Summary

Display a comparison of all trained regression models based on key evaluation metrics:

• R² Score (Train/Test)
• RMSE (Train/Test)

def safe_mape(y_true, y_pred, eps=1e-9):
    """Return MAPE (%) while ignoring zero/near-zero targets to avoid division-by-zero."""
    y_true = np.asarray(y_true, dtype=float)
    y_pred = np.asarray(y_pred, dtype=float)
    mask = np.abs(y_true) > eps
    if not np.any(mask):
        return np.nan
    return np.mean(np.abs((y_true[mask] - y_pred[mask]) / y_true[mask])) * 100.0


results = []

for name, model in models.items():
    pipe = Pipeline([
        ('preprocessor', preprocessor),
        ('regressor', model)
    ])
    pipe.fit(X_train, y_train)
    
    train_pred = pipe.predict(X_train)
    test_pred  = pipe.predict(X_test)
    
    # Core metrics
    train_r2   = r2_score(y_train, train_pred)
    test_r2    = r2_score(y_test,  test_pred)
    train_rmse = np.sqrt(mean_squared_error(y_train, train_pred))
    test_rmse  = np.sqrt(mean_squared_error(y_test,  test_pred))
    train_mae  = mean_absolute_error(y_train, train_pred)
    test_mae   = mean_absolute_error(y_test,  test_pred)
    train_mape = safe_mape(y_train, train_pred)   # in %
    test_mape  = safe_mape(y_test,  test_pred)    # in %

    # Gaps (positive => test worse than train → potential overfitting)
    r2_gap   = train_r2 - test_r2
    rmse_gap = test_rmse - train_rmse
    mae_gap  = test_mae  - train_mae
    mape_gap = test_mape - train_mape
    
    results.append({
        'Model': name,
        'Train_R2': train_r2,
        'Test_R2': test_r2,
        'R2_Gap': r2_gap,
        
        'Train_RMSE': train_rmse,
        'Test_RMSE': test_rmse,
        'Train_Gap': rmse_gap,      

        'Train_MAE': train_mae,
        'Test_MAE': test_mae,
        'MAE_Gap': mae_gap,

        'Train_MAPE': train_mape,     # %
        'Test_MAPE': test_mape,       # %
        'MAPE_Gap': mape_gap,         # %
    })

# To DataFrame (sorted by Test_R2 descending)
result_df = pd.DataFrame(results).sort_values(by='Test_R2', ascending=False).reset_index(drop=True)

# Optional: round for display
result_df_display = result_df.copy()
for col in result_df_display.columns:
    if col != 'Model':
        result_df_display[col] = result_df_display[col].astype(float).round(4)

result_df_display.head(100)

	Model	Train_R2	Test_R2	R2_Gap	Train_RMSE	Test_RMSE	Train_Gap	Train_MAE	Test_MAE	MAE_Gap	Train_MAPE	Test_MAPE	MAPE_Gap
0	RandomForest	0.9859	0.9560	0.0298	204.2587	353.1670	148.9083	115.0840	205.6813	90.5974	3.9346	8.0183	4.0837
1	XGBoost	0.9678	0.9525	0.0153	308.1696	367.2492	59.0796	208.7004	237.3678	28.6674	8.3985	9.9284	1.5298
2	DecisionTree	0.9898	0.9349	0.0549	173.2082	429.6963	256.4881	67.7856	231.8560	164.0704	1.7905	8.7402	6.9497
3	GradientBoosting	0.8892	0.8781	0.0111	571.6340	588.0019	16.3680	390.7355	401.7818	11.0463	18.1708	18.6329	0.4622
4	KNN	0.9257	0.8742	0.0516	468.1651	597.5381	129.3730	250.9227	315.9365	65.0138	10.0380	12.9934	2.9555
5	Lasso	0.6284	0.6191	0.0094	1047.0334	1039.6075	-7.4259	772.6791	768.4995	-4.1796	34.5994	34.5786	-0.0208
6	Ridge	0.6285	0.6191	0.0094	1046.9823	1039.6414	-7.3409	772.6582	768.5894	-4.0688	34.5163	34.5095	-0.0068
7	LinearRegression	0.6285	0.6191	0.0094	1046.9823	1039.6443	-7.3379	772.6611	768.5964	-4.0647	34.5149	34.5083	-0.0066

🏆 Identify the Best Performing Model

Select the best model based on:

• Highest R² on Test Set → Indicates best explanatory power
• Lowest RMSE on Test Set → Indicates best prediction accuracy

This helps determine the most reliable model for real-world application.

   
    best_model_by_r2 = result_df.loc[result_df['Test_R2'].idxmax(), 'Model']
    best_model_by_rmse = result_df.loc[result_df['Test_RMSE'].idxmin(), 'Model']

print("🏆 Best Model by R2 (filtered):", best_model_by_r2)
print("🏆 Best Model by RMSE (filtered):", best_model_by_rmse)

🏆 Best Model by R2 (filtered): RandomForest
🏆 Best Model by RMSE (filtered): RandomForest

best_model_name = 'XGBoost'

# Get the best model from the dictionary (previously selected based on evaluation metrics)
best_model = models[best_model_name]

# Build a pipeline that includes preprocessing and the chosen regression model
pipe = Pipeline([
    ('preprocessor', preprocessor),  # Apply preprocessing steps (e.g., scaling, imputing, encoding)
    ('regressor', best_model)        # Use the best regression model
])

# Train (fit) the pipeline on the training data
pipe.fit(X_train, y_train)

# Make predictions on the test set
y_pred = pipe.predict(X_test)

y_pred_train = pipe.predict(X_train)

# Calculate the residuals (errors)
residuals = y_pred - y_test
residuals_pct = (residuals / y_test) * 100  # Percentage error

# Create a DataFrame to compare actual vs predicted values
results_df = pd.DataFrame({
    
    'Actual': y_test.values,
    'Predicted': y_pred,
    'Difference ': residuals,
    'Error%': residuals_pct
})

# Show the first 5 rows of results
results_df.head()

	Actual	Predicted	Difference	Error%
14913	1500	1315.772461	-184.227539	-12.281836
19370	1200	1257.952271	57.952271	4.829356
52405	3458	4220.839844	762.839844	22.060146
6515	567	575.120972	8.120972	1.432270
7109	2600	2559.466309	-40.533691	-1.558988

# Combine test features with prediction results for detailed analysis
results_df2 = X_test.copy()  # Copy test features

# Add actual target values
results_df2['Actual'] = y_test.values

# Add predicted values
results_df2['Predicted'] = y_pred

# Calculate residuals and percentage errors
results_df2['Difference'] = results_df2['Predicted'] - results_df2['Actual']
results_df2['Error%'] = (results_df2['Difference'] / results_df2['Actual']) * 100

# Display the first 5 rows with features and prediction results
results_df2.head()


results_df2.to_excel("test_train_data13082025_01.xlsx", index=False)
 

🎯 Actual vs Predicted Plot

Visualize how closely the model's predictions align with the actual values.

plt.figure(figsize=(15, 7))
plt.scatter(results_df['Actual'], results_df['Predicted'], alpha=0.5, color='blue', label='Predictions')

# Ideal line
plt.plot(results_df['Actual'], results_df['Actual'], 'r--', label='Perfect Prediction')
 
plt.xlabel("Actual Value")
plt.ylabel("Predicted Value")
plt.title("Actual vs Predicted")
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()

⚠️ The chart below shows results from an earlier training rounds, before the latest update above.

import matplotlib.pyplot as plt
import seaborn as sns

# نفترض أن عندك y_test و y_pred
residuals = y_test - y_pred

plt.figure(figsize=(10,6))
sns.histplot(residuals, kde=True, color='blue', bins=50)
plt.axvline(0, color='red', linestyle='--', linewidth=2)
plt.xlabel("Residual Value (Actual - Predicted)")
plt.ylabel("Frequency")
plt.title("Distribution of Residuals")
plt.grid(True)
plt.show()

📊 Filtered Percentage Error Distribution (±100%)

Display a histogram of prediction errors limited to the ±100% range.
This focuses the analysis on typical prediction behavior by excluding extreme outliers.
The red dashed line at 0% represents perfect prediction.

filtered = results_df[(results_df['Error%'] >= -100) & (results_df['Error%'] <= 100)]

plt.figure(figsize=(10, 6))
plt.hist(filtered['Error%'], bins=100, color='green', edgecolor='black', alpha=0.7)
plt.axvline(0, color='red', linestyle='--', linewidth=2)
plt.xlabel("Percentage Error (%)", fontsize=12)
plt.ylabel("Count", fontsize=12)
plt.title("Distribution of % Errors (Filtered ±100%)", fontsize=14)
plt.grid(True)
plt.tight_layout()
plt.show()

📊 Distribution of Percentage Errors (Filtered ±20%)

Visualize prediction errors within the ±20% range to exclude extreme outliers.
This highlights the model’s typical performance with a clearer and more focused error distribution.

filtered = results_df[(results_df['Error%'] >= -20) & (results_df['Error%'] <= 20)]

plt.figure(figsize=(10, 6))
plt.hist(filtered['Error%'], bins=100, color='green', edgecolor='black', alpha=0.7)
plt.axvline(0, color='red', linestyle='--', linewidth=2)
plt.xlabel("Percentage Error (%)", fontsize=12)
plt.ylabel("Count", fontsize=12)
plt.title("Distribution of % Errors (Filtered ±20%)", fontsize=14)
plt.grid(True)
plt.tight_layout()
plt.show()

📈 Evaluate Model Accuracy: MAPE & MAE

• MAPE (Mean Absolute Percentage Error): Measures average percentage difference between actual and predicted values.
• MAE (Mean Absolute Error): Measures average absolute difference in the same units as the target variable.

mape = np.mean(np.abs((y_test - y_pred) / y_test)) * 100
print(f"📉 MAPE (متوسط نسبة الخطأ): {mape:.2f}%")

mae = np.mean(np.abs(y_test - y_pred))
print(f"📏 MAE (متوسط الخطأ المطلق): {mae:.2f}")

📉 MAPE (متوسط نسبة الخطأ): 9.93%
📏 MAE (متوسط الخطأ المطلق): 237.37

📊 R² Score Comparison: Train vs Test

Compare the R² scores of different models on both training and testing sets.

 
plt.figure(figsize=(14, 6))
 
bar_width = 0.35
index = np.arange(len(result_df))

# Custom colors
train_color = '#1f77b4'   # Blue
test_color = '#aec7e8'    # Light blue

plt.bar(index, result_df["Train_R2"], bar_width, label='Train R²',color=train_color)
plt.bar(index + bar_width, result_df["Test_R2"], bar_width, label='Test R²',color=test_color)

plt.xlabel("Model")
plt.ylabel("R² Score")
plt.title("R² Comparison (Train vs Test)")
plt.xticks(index + bar_width / 2, result_df["Model"], rotation=45)
plt.legend()
plt.tight_layout()
plt.show()

📉 RMSE Comparison: Train vs Test

Visualize and compare the Root Mean Squared Error (RMSE) across models for both training and testing sets.
Lower RMSE values indicate better prediction accuracy.

 
plt.figure(figsize=(14, 6))
bar_width = 0.35
index = np.arange(len(result_df))

# Custom colors
train_color = '#1f77b4'   # Blue
test_color = '#aec7e8'    # Light blue

plt.bar(index, result_df["Train_RMSE"], bar_width, label='Train RMSE',color=train_color)
plt.bar(index + bar_width, result_df["Test_RMSE"], bar_width, label='Test RMSE',color=test_color)

plt.xlabel("Model")
plt.ylabel("RMSE")
plt.title("RMSE Comparison (Train vs Test)")
plt.xticks(index + bar_width / 2, result_df["Model"], rotation=45)
plt.legend()
plt.tight_layout()
plt.show()