Merge pull request #389 from emharsha1812/main

Problems Added - 112-Cross entropy loss, 113-Early Stopping

Author: Open-Deep-ML

Problems/134_cross_entropy_loss/learn.md

@@ -0,0 +1,32 @@
+## Multi-class Cross-Entropy Loss Implementation
+
+Cross-entropy loss, also known as log loss, measures the performance of a classification model whose output is a probability value between 0 and 1. For multi-class classification tasks, we use the categorical cross-entropy loss.
+
+### Mathematical Background
+
+For a single sample with C classes, the categorical cross-entropy loss is defined as:
+
+$L = -\sum_{c=1}^{C} y_c \log(p_c)$
+
+where:
+
+- $y_c$ is a binary indicator (0 or 1) if class label c is the correct classification for the sample
+- $p_c$ is the predicted probability that the sample belongs to class c
+- $C$ is the number of classes
+
+### Implementation Requirements
+
+Your task is to implement a function that computes the average cross-entropy loss across multiple samples:
+
+$L_{batch} = -\frac{1}{N}\sum_{n=1}^{N}\sum_{c=1}^{C} y_{n,c} \log(p_{n,c})$
+
+where N is the number of samples in the batch.
+
+### Important Considerations
+
+- Handle numerical stability by adding a small epsilon to avoid log(0)
+- Ensure predicted probabilities sum to 1 for each sample
+- Return average loss across all samples
+- Handle invalid inputs appropriately
+
+The function should take predicted probabilities and true labels as input and return the average cross-entropy loss.

Problems/134_cross_entropy_loss/solution.py

@@ -0,0 +1,35 @@
+import numpy as np
+
+def compute_cross_entropy_loss(predicted_probs: np.ndarray, true_labels: np.ndarray) -> float:
+ 
+    #Given
+    epsilon = 1e-15
+    predicted_probs = np.clip(predicted_probs, epsilon, 1 - epsilon)
+    
+    #Write your code here
+    log_probs = np.log(predicted_probs)
+    loss = -np.sum(true_labels * log_probs, axis=1)
+    return float(np.mean(loss))
+
+def test_compute_cross_entropy_loss():
+    # Test case 1: Perfect predictions
+    pred1 = np.array([[1, 0, 0], [0, 1, 0]])
+    true1 = np.array([[1, 0, 0], [0, 1, 0]])
+    expected1 = 0.0
+    assert np.isclose(compute_cross_entropy_loss(pred1, true1), expected1), "Test case 1 failed"
+
+    # Test case 2: Completely wrong predictions
+    pred2 = np.array([[0.1, 0.8, 0.1], [0.8, 0.1, 0.1]])
+    true2 = np.array([[0, 0, 1], [0, 1, 0]])
+    expected2 = -np.mean([np.log(0.1), np.log(0.1)])
+    assert np.isclose(compute_cross_entropy_loss(pred2, true2), expected2), "Test case 2 failed"
+
+    # Test case 3: Typical predictions
+    pred3 = np.array([[0.7, 0.2, 0.1], [0.3, 0.6, 0.1]])
+    true3 = np.array([[1, 0, 0], [0, 1, 0]])
+    expected3 = -np.mean([np.log(0.7), np.log(0.6)])
+    assert np.isclose(compute_cross_entropy_loss(pred3, true3), expected3), "Test case 3 failed"
+
+if __name__ == "__main__":
+    test_compute_cross_entropy_loss()
+    print("All test cases passed!")

Problems/135_early_stopping/learn.md

@@ -0,0 +1,26 @@
+## Implementing Early Stopping Criterion
+
+Early stopping is a regularization technique that helps prevent overfitting in machine learning models. Your task is to implement the early stopping decision logic based on the validation loss history.
+
+### Problem Description
+
+Given a sequence of validation losses from model training, determine if training should be stopped based on the following criteria:
+
+- Training should stop if the validation loss hasn't improved (decreased) for a specified number of epochs (patience)
+- An improvement is only counted if the loss decreases by more than a minimum threshold (min_delta)
+- The best model is the one with the lowest validation loss
+
+### Example
+
+Consider the following validation losses: [0.9, 0.8, 0.75, 0.77, 0.76, 0.77, 0.78]
+
+- With patience=2 and min_delta=0.01:
+  - Best loss is 0.75 at epoch 2
+  - No improvement > 0.01 for next 2 epochs
+  - Should stop at epoch 4
+
+### Function Requirements
+
+- Return both the epoch to stop at and the best epoch
+- If no stopping is needed, return the last epoch
+- Epochs are 0-indexed

Problems/135_early_stopping/solution.py

@@ -0,0 +1,50 @@
+import numpy as np
+from typing import Tuple
+
+def early_stopping(val_losses: list[float], patience: int, min_delta: float) -> Tuple[int, int]:
+    
+    best_loss = float('inf')
+    best_epoch = 0
+    epochs_without_improvement = 0
+    
+    for epoch, loss in enumerate(val_losses):
+        # Check if current loss is better than best loss by at least min_delta
+        if loss < best_loss - min_delta:
+            best_loss = loss
+            best_epoch = epoch
+            epochs_without_improvement = 0
+        else:
+            epochs_without_improvement += 1
+            
+        # Check if we should stop
+        if epochs_without_improvement >= patience:
+            return epoch, best_epoch
+            
+    # If we never hit the patience threshold, return the last epoch
+    return len(val_losses) - 1, best_epoch
+
+def test_early_stopping():
+    
+    losses1 = [0.9, 0.8, 0.75, 0.77, 0.76, 0.77, 0.78]
+    stop_epoch1, best_epoch1 = early_stopping(losses1, patience=2, min_delta=0.01)
+    assert stop_epoch1 == 4 and best_epoch1 == 2, "Test case 1 failed"
+
+    losses2 = [0.9, 0.8, 0.7, 0.6, 0.5]
+    stop_epoch2, best_epoch2 = early_stopping(losses2, patience=2, min_delta=0.01)
+    assert stop_epoch2 == 4 and best_epoch2 == 4, "Test case 2 failed"
+
+    losses3 = [0.9, 0.8, 0.79, 0.78, 0.77]
+    stop_epoch3, best_epoch3 = early_stopping(losses3, patience=2, min_delta=0.1)
+    assert stop_epoch3 == 4 and best_epoch3 == 2, "Test case 3 failed"
+
+    losses4 = [0.5, 0.4]
+    stop_epoch4, best_epoch4 = early_stopping(losses4, patience=3, min_delta=0.01)
+    assert stop_epoch4 == 1 and best_epoch4 == 1, "Test case 4 failed"
+
+    losses5 = [0.5, 0.4, 0.4, 0.4, 0.4]
+    stop_epoch5, best_epoch5 = early_stopping(losses5, patience=2, min_delta=0.01)
+    assert stop_epoch5 == 3 and best_epoch5 == 1, "Test case 5 failed"
+
+if __name__ == "__main__":
+    test_early_stopping()
+    print("All test cases passed!")

Problems/problem_list.py

@@ -0,0 +1,22 @@
+import os
+
+problems_folder = "Problems"
+folder_names = []
+
+# Check if the "Problems" folder exists
+if os.path.exists(problems_folder) and os.path.isdir(problems_folder):
+    # Iterate through all items in the "Problems" folder
+    for item in os.listdir(problems_folder):
+        # Check if the item is a directory
+        item_path = os.path.join(problems_folder, item)
+        if os.path.isdir(item_path):
+            folder_names.append(item)
+
+    # Write the folder names to a text file
+    with open("problems_folder_list.txt", "w") as f:
+        for name in folder_names:
+            f.write(name + "\n")
+
+    print("Folder names inside 'Problems' written to problems_folder_list.txt")
+else:
+    print("The 'Problems' folder does not exist.")

folder_list.txt

@@ -0,0 +1,5 @@
+.git
+.github
+example_problem
+Problems
+__pycache__

problems_folder_list.txt

@@ -0,0 +1,93 @@
+100_Softsign
+102_Swish
+103_SELU
+104_logistic_regression
+105_train_softmaxreg
+106_train_logreg
+107_masked_attention
+10_Calculate_cov_matrix
+110_METEOR
+111_PMI
+112_cross_entropy_loss
+113_early_stopping
+11_Linear_equations_jacobi
+12_svd_2x2_singular_values
+13_determinant_4x4
+14_linear_regression_normal_equation
+15_linear_regression_gradient_descent
+16_feature_scaling
+17_k_means_clustering
+18_cross_validation_split
+19_PCA
+1_matrix_times_vector
+20_decision_tree_learning
+21_pegasos_kernel_svm
+22_sigmoid
+23_softmax
+24_single_neuron
+25_Single_Neuron_with_Backpropagation
+26_Autograd
+28_svd_2x2_eign
+2_transpose_matrix
+32_polynomial_features
+33_random_subsets
+35 - Vector to Matrix
+39_log_softmax
+3_reshape_matrix
+41_simple_conv2d_layer
+42_relu_activation_function
+43_ridge_loss
+44_leaky_relu
+45_linear_kernel
+46_precision
+47_gradient_descent
+48_rref
+4_calculate_mean_by_row_or_column
+50_lasso_regression_gradient_descent
+51_OSA_distance
+52_recall
+53_self_attention
+54_RNN_layer
+55_2D_translation_matrix
+57_gauss_seidel
+58_gaussian_elimination_partial_pivoting
+5_scalar_multiplication_of_a_matrix
+60_TF-IDF
+61_f_score
+63_conjugate_gradient
+64_gini_impurity
+65_compressed_row_sparse_matrix
+66_orthogonal_projection
+67_compressed_column_sparse_matrix
+69_r-squared
+6_calculate_eigenvalues
+70_image_brightness_calculator
+71_Room_Mean_Square_Error
+73_Dice_Score
+74_hdc_bundling_binding
+75_classification_performance_metrics
+76_Cosine_Similarity
+77_classification_performance_metrics
+78_binomial_distribution
+78_descriptive_stats
+7_transform_matrix
+80_normal_distribution
+81_poisson_distribution
+82_image_basic_contrast_calculator
+83_vector_dot_product
+84_phi_transformation
+85_Positional_encoding
+86_overfitting_underfitting
+87_adam_optimizer
+88_GPT_2
+8_Calculate_2x2_Matrix_Inverse
+94_multi_head_attention
+95_phi_correlation_coefficient
+96_Hard_Sigmoid
+97_ELU
+98_PReLU
+99_Softplus
+9_matrixmul
+BM25
+interactive_learn
+MAE