The perceptron is a fundamental concept in deep learning, with many algorithms stemming from its original design.
In this tutorial, I’ll show you how to build both single layer and multi-layer perceptrons (MLPs) across three frameworks:
Custom classifier
Scikit-learn’s MLPClassifier
Keras Sequential classifier using SGD and Adam optimizers.
This will help you learn about their various use cases and how they work.
Table of Contents
What is a Perceptron?
How to Build a Single-Layered Classifier
What is a Multi-Layer Perceptron?
How to Build Multi-Layered Perceptrons
Understanding Optimizers
How to Build an MLP Classifier with SGD Optimizer
How to Build an MLP Classifier with Adam Optimizer
Final Results: Generalization
Conclusion
Prerequisites
Mathematics (Calculus, Linear Algebra, Statistics)
Coding in Python
Basic understanding of Machine Learning concepts
What is a Perceptron?
A perceptron is one of the simplest types of artificial neurons used in Machine Learning. It’s a building block of artificial neural networks that learns from labeled data to perform classification and pattern recognition tasks, typically on linearly separable data.
A single-layer perceptron consists of a single layer of artificial neurons, called perceptrons.
But when you connect many perceptrons together in layers, you have a multi-layer perceptron (MLP). This lets the network learn more complex patterns by combining simple decisions from each perceptron. And this makes MLPs powerful tools for tasks like image recognition and natural language processing.
The perceptron consists of four main parts:
Input layer: Takes the initial numerical values into the system for further processing.
Weights: Combines input values with weights (and bias terms).
Activation function: Determines whether the neuron should fire based on the threshold value.
Output layer: Produces classification result.
It performs a weighted sum of inputs, adds a bias, and passes the result through an activation function – just like logistic regression. It’s sort of like a little decision-maker that says “yes” or “no” based on the information it gets.
So for instance, when we use a sigmoid activation, its output is a probability between 0 and 1, mimicking the behavior of logistic regression.
Applications of Perceptrons
Perceptrons are applied to tasks such as:
Image classification: Perceptrons classify images containing specific objects. They achieve this by performing binary classification tasks.
Linear regression: Perceptrons can predict continuous outputs based on input features. This makes them useful for solving linear regression problems.
How the Activation Function Works
For a single perceptron used for binary classification, the most common activation function is the step function (also known as the threshold function):
$$\phi(z) = \begin{cases} 1 &\text{if } z \geq \theta \\ \\ 0 &\text{if } z < \theta \end{cases}$$
where:
ϕ(z): the output of the activation function.
z: the weighted sum of the inputs plus the bias:
$$z = \sum_{i=1}^m w_i x_i + b$$
(xi: input values, w: weight associated with each input, b: bias terms)
θ is the threshold. Often, the threshold θ is set to zero, and the bias (b) effectively controls the activation threshold.
In that case, the formula becomes:
$$\phi(z) = \begin{cases} 1 &\text{if } z \geq 0 \\ \\ 0 &\text{if } z < 0 \end{cases}$$
When the step function ϕ(z) outputs one, it signifies that the input belongs to the class labeled one.
This occurs when the weighted sum is greater than zero, leading the perceptron to predict the input is in this binary class.
While the step function is conceptually the original activation for a perceptron, its discontinuity at zero causes computational challenges.
In modern implementations, we can use other activation functions like the sigmoid function:
$$\sigma (z) = \frac {1} {1 + e^{-z}}$$
The sigmoid function also outputs zero or one depending on the weighted sum (z).
How the Loss Function Works
The loss function is a crucial concept in machine learning that quantifies the error or discrepancy between the model's predictions and the actual target values.
Its purpose is to penalize the model for making incorrect or inaccurate predictions, which guides the learning algorithm (for example, gradient descent) to adjust the model's parameters in a way that minimizes this error and improves performance.
In a binary classification task, the model may adopt the hinge loss function to penalize misclassifications by incurring an additional cost for incorrect predictions:
$$L(y, h(x)) = max(0, 1- y*h(x))$$
(h(x): prediction label, y: true label)
How to Build a Single-Layered Classifier
Now, let’s build a simple single-layer perceptron for binary classification.
1. Custom Classifier
Initialize the classifier
We’ll first initialize the classifier with weights, bias, number of epochs (n_iterations), and learning_rates.
Define the activation function
Use a step function that returns zero if input (x) ≤ 0, else 1. By default, the threshold is set to zero.
Train the model
Now it’s time to start training. The learning process involves iteratively updating the perceptron’s internal parameters: weights and bias.
This process is controlled by a specified number of training epochs defined by n_iterations.
In each epoch, the model processes the entire input dataset (X) and adjusts its weights and bias based on the difference between its predictions and the true labels (y), guided by a predefined learning_rate.
How the weights work in the iteration loop
The weights in a perceptron define the orientation (slope) of the decision boundary that separates the classes.
Its iterative update in the for loop aims to reduce classification errors such that:
$$\begin {align*} w_j &:= w_j + \Delta w_j \\ & := w_j + \eta (y_i - \hat y_i)x_{ij} \\ &= \begin{cases} w_j &\text{(a) } y_i - \hat y_i = 0\\ w_j + \eta x_ij &\text{(b) } y_i - \hat y_i = 1 \\ w_j - \eta x_ij &\text{(c) } y_i - \hat y_i = -1 \\ \end{cases} \end{align*}$$
(w_j: j-th weight, η: learning rate, (yi−y^i): error)
This means that:
When the prediction is correct, the error is zero, so the weight is unchanged.
When the prediction is too low (yi=1 and y^i=0), the weight is adjusted to the same direction to increase the weighted sum.
When the prediction is too high (yi=0 and y^i=1), the weight is adjusted to the opposite direction to pull the weighted sum lower.
How the bias terms work in the iteration loop
The bias determines the decision boundary’s intercept (position from the origin).
Similar to weights, we adjust the bias terms in each epoch to position the decision boundary:
$$\begin {align*} b &:= b + \Delta b \\ & := b + \eta (y_i - \hat y_i) \\ &= \begin{cases} b &\text{(a) } y_i - \hat y_i = 0\\ b + \eta &\text{(b) } y_i - \hat y_i = 1 \\ b - \eta &\text{(c) } y_i - \hat y_i = -1 \\ \end{cases} \end{align*}$$
This repeated adjustment aims to optimize the model’s ability to correctly classify the training data.
Make a prediction
Lastly, we add a function to generate an outcome value (zero or one) for a new, unseen data (X):
The entire classifier looks like this:
Simulate with synthetic datasets
First, we generated a synthetic linearly separable dataset using make_blob and computed a decision boundary, then train the classifier we created.
Results
The classifier generated a clear, highly accurate linear decision boundary.
Accuracy (Train): 0.981
Accuracy (Test): 0.975
2. Leverage SckitLearn’s MCP Classifier
For our convenience, we’ll use sckit-learn’s build-in classifier ( MCPClassifier) to build a similar, yet more robust classifier:
Results
The MCP Classifier generated a clear linear decision boundary with slightly better accuracy scores.
Accuracy (Train): 0.985
Accuracy (Test): 0.995
Limitations of Single-Layer Perceptrons
Now, let’s talk about the key differences between the MCP Classifier and our custom single-layer perceptron.
Unlike more general neural networks, single-layer perceptrons use a step function as their activation.
Due to its discontinuity at x=0, the step function is not differentiable over its entire domain (−∞ to ∞).
This fundamental property precludes the use of gradient-based optimization algorithms such as SGD or Adam, as these methods depend on the computation of gradients, partial derivatives for the cost function.
In contrast, most neural networks employ differentiable activation functions (for example, sigmoid, ReLU) and loss functions (for example, MSE, Cross-Entropy) for effective optimization.
Other challenges of a single-layer perceptron include:
Limited to linear separability: Because they can only learn linear decision boundaries, they are unable to handle complex, non-linearly separable data.
Lack of depth: Being single-layered, they cannot learn complex hierarchical representations.
Limited optimizer options: As mentioned, their non-differentiable activation function precludes the use of major gradient-based optimizers.
So, in the next section, you’ll learn about multi-layered perceptrons to overcome the disadvantages.
What is a Multi-Layer Perceptron?
An MLP is a class of feedforward artificial neural network that consists of at least three layers of nodes:
an input layer,
one or more hidden layers, and
an output layer.
Except for the input nodes, each node is a neuron that uses a nonlinear activation function.
MLPs are widely used for classification problems as well as regression:
Classification tasks: MLPs are widely used for classification problems, such as handwriting recognition and speech recognition.
Regression analysis: They are also applied in regression problems where the relationship between input and output is complex.
How to Build Multi-Layered Perceptrons
Let’s handle a binary classification task using a standard MLP architecture.
Outline of the Project
Objective
Detect fraudulent transactions
Evaluation Metrics
Considering the cost of misclassification, we’ll prioritize improving Recall and Precision scores
Then check the accuracy of classification with Accuracy Score (TP + TN / (TP + TN + FP + FN ))
Cost of Misclassification (from high to low):
False Negative (FN): The model incorrectly identifies a fraudulent transaction as legitimate (Missing actual fraud)
False Positive (FP): The model incorrectly identifies a legitimate transaction as fraudulent (Blocking legitimate customers.)
True Positive (TP): The model correctly identifies a fraudulent transaction as fraud.
True Negative (TN): The model correctly identifies a non-fraudulent transaction as non-fraud.
Planning an MLP Architecture
In the network, 19 input features feed into the first hidden layer’s 30 neurons, which use a ReLU activation function.
Then, their outputs are passed to the second layer, culminating in sigmoid values as the final output.
During the optimization process, we’ll let the optimizer (SGD and Adam) perform forward and backward passes to adjust parameters.
Image: Standard MLP Architecture for Binary Classification Tasks (Created by Kuriko Iwai using image source)
Especially in deeper network, ReLU is advantageous in preventing vanishing gradient problems where gradients become extremely small as they are backpropagated from the output layers.
Learn More: A Comprehensive Guide on Neural Network in Deep Learning
Preprocessing the Datasets
First, we consolidate three datasets – transaction, customer, and credit card – into a single DataFrame, independently sanitizing numerical and categorical data:
DataFrame:
Our DataFrame shows an extremely skewed data distribution with:
Fraud samples: 1,191
Non-fraud samples: 11,477,397
For classification tasks, it's crucial to be aware of sample size imbalances and employ appropriate strategies to mitigate their negative impact on classification model performance, especially regarding the minority class.
For our data, we’ll:
split the 1,191 fraud samples into training, validation, and test sets,
add an equal number of randomly chosen non-fraud samples from the DataFrame, and
adjust split balances later if generalization challenges arise.
After the operation, we secured 1,582 training, 400 validation, and 400 test samples, each dataset maintaining a 50:50 split between fraud and non-fraud transactions:
Considering the high dimensional feature space with 19 input features, we’ll apply SMOTE to resample the training data (SMOTE should not be applied to validation or test sets to avoid data leakage):
We’ve secured 4,000 training samples, maintaining a 50:50 split between fraud and non-fraud transactions:
Lastly, we’ll apply column transformers to numerical and categorical features separately.
Column transformers are advantageous in handling datasets with multiple data types, as they can apply different transformations to different subsets of columns while preventing data leakage.
Understanding Optimizers
In deep learning, an optimizer is a crucial element that fine-tunes a neural network’s parameters during training. Its primary role is to minimize the model’s loss function, enhancing performance.
Various optimization algorithms, known as optimizers, employ distinct strategies to converge towards optimal parameters for improved predictions efficiently.
In this article, we’ll use the SGD Optimizer and Adam Optimizer.
1. How a SGD (Stochastic Gradient Descent) Optimizer Works
SGD is a major optimization algorithm that computes the gradient (partial derivative of the cost function) using a small mini-batch of examples at each epoch:
$$\begin{align*} w_j &:= w_j - \eta \frac {\partial J} {\partial w_j} \\ \\ b &:= b - \eta \frac {\partial J} {\partial b} \end{align*}$$
(w: weight, b: bias, J: cost function, η: learning rate)
In binary classification, the cost function (J) is defined with a sigmoid function (σ(z)) where z generates weighted sum of inputs and bias terms:
$$\begin{align*} J(y, \hat y) &=−[y log(\hat y) + (1-y)log(1-\hat y)] \\ \\ \hat y &= \sigma (z) = \frac {1} {1+e^{-z}} \\ \\ z &= \sum_{i=1}^m w_i x_i + b \end {align*}$$
2. How Adam (Adaptive Moment Estimation) Optimizer Works
Adam is an optimization algorithm that computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients.
Adam optimizer combines the advantages of RMSprop (using squared gradients to scale the learning rate) and Momentum (using past gradients to accelerate convergence):
$$w_{j,t+1} = w_{j,t} - \alpha \cdot \frac{\hat{m}{t,w_j}}{\sqrt{\hat{v}{t,w_j}} + \epsilon}$$
where:
α: The learning rate (default is 0.001)
ϵ: A small positive constant used to avoid division by zero
m^: First moment (mean) estimate with a bias correction, leveraging Momentum:
$$\begin{align*} \hat m_t &= \frac {m_t} {1 - \beta_1^t} \\ \\ m_t &= \beta_1 m_{t-1} + (1-\beta_1) \underbrace{ \frac {\partial L} {\partial w_t}}_{\text{gradient}} \end{align*}$$
(β1: Decay rates, typically set to β1=0.9)
v^: Second moment (variance) estimate with a bias correction, leveraging RMSprop:
$$\begin{align*} \hat v_t &= \frac {v_t} {1 - \beta_2^t} \\ \\ v_t &=\beta_2 v_{t-1} + (1- \beta_2) (\frac {\partial L} {\partial w_t})^2 \end {align*}$$
(β2: Decay rates, typically set to β2=0.999)
Since both m and v are initialized at zero, Adam computes the bias-corrected estimates to prevent them being biased toward zero.
Learn More: A Comprehensive Guide on Neural Network in Deep Learning
How to Build an MLP Classifier with SGD Optimizer
Custom Classifier
This process involves a forward pass and backpropagation, during which SGD computes optimal weights and biases using gradients:
In the process of the forward pass, the network calculates a weighted sum of weights and bias (z), applies an activation function (ReLU) to the values in each hidden layer, and then computes the predicted output (y_pred) using a sigmoid function.
So the final classifier looks like this:
Training / Prediction
Train the model and make a prediction using training and validation datasets:
Results
Recall: 0.7930 — 0.6650 (from training to validation)
Precision: 0.7790 — 0.6786 (from training to validation)
The model effectively learned and generalized the patterns, achieving a Recall of 79.3% (approximately 80% accuracy in identifying fraud transactions) with a 12-point drop on the validation set.
Loss history:
We visualized the decision boundary using the first two principal components (PCA) as the x and y axes. Note that the boundary is non-linear.
Leverage SckitLearn’s MCP Classifier
We can use an MCP Classifier to define a similar model, incorporating;
Early stopping using internal validation to prevent overfitting and
L2 regularization with a small tolerance.
Results
Recall: 0.7830 - 0.6200 (from training to validation)
Precision: 0.8208 - 0.6703 (from training to validation)
The model showed strong performance during training, achieving a Recall of 78.30%. Its performance declined on the validation set.
This suggests that while the model learned effectively from the training data, it may be overfitting and not generalizing as well to unseen data.
Leverage Keras Sequential Classifier
For the sequential classifier, we can further enhance the classifier by:
Initializing the output layer’s bias with the log-odds of positive class occurrences in the training data (y_train) to address dataset imbalance and promote faster convergence,
Integrating 10% dropout between hidden layers to prevent overfitting by randomly deactivating neurons during training,
Including Precision and Recall in the model’s compilation metrics to optimize for classification performance,
Applying class weights to penalize misclassifications of the minority class more heavily, improving the model’s ability to learn rare patterns, and
Utilizing a separate validation dataset for monitoring performance during training to help detect overfitting and guides hyperparameter tuning.
Results
Recall: 0.7125 — 0.7250 (from training to validation)
Precision: 0.7607 — 0.7545 (from training to validation)
Given that the gaps between training and validation are relatively small, the model is generalizing reasonably well.
It suggests that the regularization techniques are likely effective in preventing significant overfitting.
How to Build an MLP Classifier with Adam Optimizer
Custom Classifier
This iterative process of updating parameters occurs within the mini-batch loop to keep updating weights and bias:
Following the principles of forward and backward passes, we construct the final classifier by initializing it with beta1 and beta2, built upon an MLP_SGD architecture:
Training / Prediction
Train the model and make a prediction using training and validation datasets:
Results
Recall: 0.9870–0.6150 (from training to validation)
Precision: 0.9811–0.6474 (from training to validation)
While the Adam optimizer outperformed SGD, the model exhibited significant overfitting, with both Recall and Precision falling by around 30 points between training and validation.
Loss History
We visualized the decision boundary using the first two principal components (PCA) as the x and y axes.
Leverage SckitLearn’s MCP Classifier
We’ve switched the optimizer from SGD to Adam, keeping all other settings constant:
Results
Recall: 0.8975–0.6400 (from training to validation)
Precision: 0.8864 — 0.6305 (from training to validation)
Despite a performance improvement compared to the SGD optimizer, the significant drop in both Recall (from 0.8975 to 0.6400) and Precision (from 0.8864 to 0.6305) from training to validation data indicates that the model is still overfitting.
Leverage Keras Sequential Classifier
Similar to MLPClassifier, we’ve switched the optimizer from SGD to Adam with all the other conditions remaining the same:
Results
Recall: 0.7995–0.7500 (from training to validation)
Precision: 0.8409–0.8065 (from training to validation)
The model exhibits good performance, with Recall slightly decreasing from 0.7995 (training) to 0.7500 (validation), and Precision similarly dropping from 0.8409 (training) to 0.8065 (validation).
This indicates good generalization, with only minor performance degradation on unseen data.
Final Results: Generalization
Finally, we’ll evaluate the model’s ultimate performance on the test dataset, which has remained completely separate from all prior training and validation processes.
Overall, the Keras Sequential model, optimized with SGD, achieved the best performance with an AUPRC (Area Under Precision-Recall Curve) of 0.72.
Conclusion
In this exploration, we experimented with custom classifiers, Scikit-learn models, and Keras deep learning architectures.
Our findings underscore that effective machine learning hinges on three critical factors:
robust data preprocessing (tailored to objectives and data distribution),
judicious model selection, and
strategic framework or library choices.
Choosing the right framework
Generally speaking, choose MLPClassifier when:
You’re primarily working with tabular data,
You want to prioritize simplicity, quick iteration, and seamless integration,
You have simple, shallow architectures, and
You have a moderate dataset size (manageable on a CPU).
Choose Keras Sequential when:
You’re dealing with image, text, audio, or other sequential data,
You’re building deep learning models such as CNNs, RNNs, LSTMs,
You need fine-grained control over the model architecture, training process, or custom components,
You need to leverage GPU acceleration,
You’re planning for production deployment, and
You want to experiment with more advanced deep learning techniques.
Limitation of MLPs
While Multilayer Perceptrons (MLPs) proved valuable, their susceptibility to computational complexity and overfitting emerged as key challenges.
Looking ahead, we’ll delve into how Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) offer powerful solutions to these inherent MLP limitations.
You can find more info about me on my Portfolio / LinkedIn / Github.