Simple Nerural Net

In a windows computer a folder with the name of the project  "StudentPerformance" will be created under c:\Users\<username>\pycharm Projects\StudentPerformance.

Within the project create 3 python files named:

1. student_performance.py

In this file we will create the model, load the training data, train the model and save it for future usage.

2. use_student_performance.py

In this file will write a sample python program that loads the saved model and uses it 

3. student_performance_api.py

In this file we will publish the model in a fast api, accessible to be accessed from a network sockect (IP:Port)

all packages are installed inside your project’s virtual environment folder, .venv folder

import numpy as np

import pandas as pd

import tensorflow as tf

from keras.src.layers import Dense

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import LabelEncoder

from tensorflow import keras

from tensorflow.keras.initializers import Zeros

# 1. Load dataset from CSV (make sure it's in your project folder or data/ subfolder)

df = pd.read_csv("data/student_training_data.csv")

#print(df.head())

# 2. Encode categorical variables

le_gender = LabelEncoder()

le_city = LabelEncoder()

le_perf = LabelEncoder()

#print (le_gender,le_city,le_perf)

df['Gender'] = le_gender.fit_transform(df['Gender'])

df['City'] = le_city.fit_transform(df['City'])

df['Performance'] = le_perf.fit_transform(df['Performance'])

#print(df.head())

X=df[["Gender","HighSchoolGrade","City"]].values

Y=df["Performance"].values

#print(X[0])

#print(Y[0])

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

#print(X_train.shape)

#print(X_test.shape)

#print(Y_train.shape)

#print(Y_test.shape)

#print(X_train[0])

#print(Y_train[0])

model = tf.keras.models.Sequential()

model.add(Dense(8, input_shape=(3,), activation="relu",   bias_initializer=Zeros(), name="input_hidden"))

model.add(Dense(6, activation="relu", bias_initializer=Zeros(),name="hidden_layer_2"))

model.add(Dense(4, activation="relu",  bias_initializer=Zeros(), name="hidden_layer_3"))

model.add(Dense(3, activation="softmax",  bias_initializer=Zeros(), name="output_layer"))

model.compile(

   optimizer="adam",

   loss="sparse_categorical_crossentropy",

   metrics=["accuracy"]

)

print("\n Model Summary: \n")

model.summary()

# Print initial weights (before training)

for layer in model.layers:

   weights = layer.get_weights()

   print(f"\nLayer {layer.name} initial weights:")

   for w in weights:

       print(w.shape)

       print(w)

history = model.fit(X_train, Y_train, epochs=100, batch_size=16)

# Print  weights (after training)

for layer in model.layers:

   weights = layer.get_weights()

   print(f"\nLayer {layer.name} alter trainng weights:")

   for w in weights:

       print(w.shape)

       print(w)

loss, acc = model.evaluate(X_test, Y_test,verbose=0)

print("\nTest loss:", loss)

print("Test accuracy:", acc)

#example prediction

sample = pd.DataFrame([{

   "Gender": "Female",

   "HighSchoolGrade": 9,

   "City": "Tirana"

}])

sample["Gender"] = le_gender.transform(sample["Gender"])

sample["City"] = le_city.transform(sample["City"])

prediction = model.predict(sample.values)

pred_class = np.argmax(prediction)

print("Predicted Performance:", le_perf.inverse_transform([pred_class])[0])

# Save the trained model

model.save("student_model.keras")

# Save the encoders separately using joblib

import joblib

joblib.dump(le_gender, "le_gender.pkl")

joblib.dump(le_city, "le_city.pkl")

joblib.dump(le_perf, "le_perf.pkl")

print(" Model and encoders saved in project folder.")

import pandas as pd

import numpy as np

import tensorflow as tf

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import LabelEncoder

from tensorflow import keras

from tensorflow.keras.initializers import Zeros

       HighSchoolGrade

count       500.000000

mean          7.462000

std           1.715354

min           5.000000

25%           6.000000

50%           8.000000

75%           9.000000

max          10.000000

The complete code to encode our training data:

print('Original Data: \n')

print(df.head())

# 2. Encode categorical variables

# 2.a create label encoder objects for each of the text coluns

le_gender = LabelEncoder()

le_city = LabelEncoder()

le_perf = LabelEncoder()

#2.b encode values

df['Gender'] = le_gender.fit_transform(df['Gender'])

df['City'] = le_city.fit_transform(df['City'])

df['Performance'] = le_perf.fit_transform(df['Performance'])

print('Encoded Data: \n')

print(df.head())

print commands show the original data and the encoded data:

Original Data: 

 Gender  HighSchoolGrade     City Performance

0    Male                9  Shkodra   excellent

1  Female                9   Tirana   excellent

2    Male                5   Durres         bad

3    Male                7   Durres        good

4    Male                6   Durres         bad

Encoded Data: 

Gender  HighSchoolGrade  City  Performance

0       1                9     1            1

1       0                9     2            1

2       1                5     0            0

3       1                7     0            2

4       1                6     0            0

model = tf.keras.models.Sequential()

model.add(Dense(8, input_shape=(3,), activation="relu",   bias_initializer=Zeros(), name="input_hidden"))

model.add(Dense(6, activation="relu", bias_initializer=Zeros(),name="hidden_layer_2"))

model.add(Dense(4, activation="relu",  bias_initializer=Zeros(), name="hidden_layer_3"))

model.add(Dense(3, activation="softmax",  bias_initializer=Zeros(), name="output_layer"))

here 

model = tf.keras.models.Sequential()

creates an empty sequential model.

To the model layers are added sequentially using model.add(layer)

In this example we use Dense layers, which mean that it is connected to all nodes on the other layer. 

It means each neuron in this layer receives input from every neuron in the previous layer 

Mathematicaly a dense layer performs:

y=f(Wx+b) 

where 

• W = weight matrix (learned multipliers)

• x = input vector (your features)

• b = bias vector

• f = activation function (e.g., ReLU)

Dense() method

The Dense method creates a layer of neurons, 

It takes as input parameters:

• number of neurons (units), example the input layer has 8 neurons

• input_shape( x,) , example input_shape(3,) means the the neuron expects inputs of 3 variables (x1,x2,x3), in our example (gender, High School Grade, City). We fix only one dimension while it can receive a variable number of records of 3 variables.

• activation function, ex. activation="relu" or activation="softmax". These are the activation functions that control the value returned by the neurons of the layer. 

  • relu : Rectified Linear Unifier, is the function , f(x)=max(0,x)  , so it returns either the computed value is it is positive or 0 if it is negative.  ReLU (Rectified Linear Unit) helps networks learn non-linear patterns while avoiding vanishing gradients. 

  • softmax :  is one of the most important functions in machine learning, especially for classification problems like your student performance prediction (bad / good / excellent). 
    We use it in the output layer, which has 3 neurons, one for each class: bad, good, excellent.

• kernel_initializer

kernel_initializer defines how the weight vector of the layer will be initialized. 

The weights can be initialized to:

• Zeros(), 

• glorot_uniform (default), 

• random_normal, 

• uniform, 

• he_normal 

• bias_initializer

It defines the initial values for the bias vector b.  Default is usualy Zeros().

• name , a name given to the layer, optional.

compile() configures the model for training.

It tells Tensorflow three things:

1. How to update weights ( the optimizer)

2. How to measure errors (the loss function)

3. How to report performance (metrics)

Let's briefly explain each of the items used in our model:

Optimizer

optimizer="adam" 

The optimizer controls how the model weights are updated after each training step.

Adam is short for Adaptive Moment Estimation, is one of the most popular optimizers. 

It combines the strength of two classical methods:

• Momentum - remembers previous gradient

• RMSProp - scales learning rate based on recent gradient magnitude

It automaticaly adjusts learning rates for each parameter, providing faster more stable convergence. 

Loss function

the loss function tells the model how to measure how wrong it is after each prediction. The optimizer tries to minimize this loss value. Since we are doing multi-class classification ( 3 output neurons, one correct class) , we use cross_entropy, which measures the distance between the true label and the predicted probability distribution. 

We can use categorical_crossentropy or sparse_categorical_crossentropy.

The main difference is the exptected label format:  

• categorical_crossentropy expect a one-hot encoded vector for each label, such as [0,1,0] or [1,0,0]

• sparse_categorical_crossentropy expects an integer encoded labels, such as 0 or 1 or 2 

Before with label encoder we encoded each label as an integer value herewith we will use the sparse_categorical_crossentropy

metrics

Metrics are for monitoring the model, they do not influence training. 

During training keras will display both:

• loss - tha value being optimized

• accuracy - % of predictions where the highest probability class equals the true class

Now we have created the model. we compiled it, we hve prepared the training data, so we are ready to train the model.

We train the model using the model.fit() method:

history = model.fit(

   X_train,         # your input features

   Y_train,         # putput value

   epochs=50,       # how many times to go through the full dataset

   batch_size=16,   # how many samples per training step

   validation_split=0.2,  # optional: keep 20% for validation

   verbose=1        # show progress bar

)

Let us explain the parameters:

X_train: contains the input data, in our example is the array containing [ Gender, Grade, City ]

Y_train: contains the student performance for each of the input records.  This value is known in the training data. The loss function is calculated as a difference between this true value and the predicted value. The optimizer tries to optimize this value.

batch_size:  is the number of the records that the model processes before updating the weights. In our case it predicts 16 cases (batch size) before it adjusts the weights. Usually, loss is calculated as a mean of losses of each individual case.

epochs: an epoch is a full pass over all the dataset. epochs are the number of times to go over all the dataset

verbose: show the progress bar. 1= show, 0= do not show

validation_split:  what amount of data is used for validation. In our case 20%, validation_split=0.2

model.fit() returns the history of training metrics per each epoch

What is validation data? 

It's a holdout set from your training data, which your model doesn't use to adjust weights, but it's still used to measure the performance after each epoch.

So during the training: 

the model learns (adjusts weights) using 80% of the data

after each epoch it tests itself on validation data (20%)

it reports both at the end of each epoch:

loss:...., accuracy:..., val_loss:...., val_accuracy:....

The validation set helps detect:

• Overfitting, when training accuracy but validation accuracy worsens

• Underfitting, when both are lows

Why we need validation 

 Because training accuracy alone can be misleading.

A model can learn to perfectly memorize the training data, but fail on new unseen data. 

Validation data simulates unseen data- it tells you if the model generalizes well.

you can visualize the result of the training usung matplotlib.

Before in the python terminal within pycharm you need to install matplotlib.

>pip install matplotlib

then import it in the beging of the python file

import matplotlib as plt

than after the fit command, that stores the training result in history object.

history = model.fit(

   X_train,         # your input features

   Y_train,         # putput value

   epochs=50,       # how many times to go through the full dataset

   batch_size=16,   # how many samples per training step

   validation_split=0.2,  # optional: keep 20% for validation

   verbose=1        # show progress bar

)

use the matplot lib to visualize it

plt.plot(history.history['accuracy'], label='train accuracy')

plt.plot(history.history['val_accuracy'], label='validation accuracy')

plt.xlabel('Epoch')

plt.ylabel('Accuracy')

plt.legend()

plt.show()

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step - accuracy: 0.8300 - loss: 0.7141  

.idea

.venv

data

le_city.pkl

le_gender.pkl

le_perf.pkl

student_model.keras

student_predictor.py