Merge pull request #500 from Open-Deep-ML/new-q-157

added new Q

Author: Open-Deep-ML

questions/157_implement-the-bellman-equation-for-value-iteration/description.md

@@ -0,0 +1 @@
+Write a function that performs one step of value iteration for a given Markov Decision Process (MDP) using the Bellman equation. The function should update the state-value function V(s) for each state based on possible actions, transition probabilities, rewards, and the discount factor gamma. Only use NumPy.

questions/157_implement-the-bellman-equation-for-value-iteration/example.json

@@ -0,0 +1,5 @@
+{
+  "input": "import numpy as np\ntransitions = [\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, False)]},\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, True)]}\n]\nV = np.array([0.0, 0.0])\ngamma = 0.9\nnew_V = bellman_update(V, transitions, gamma)\nprint(np.round(new_V, 2))",
+  "output": "[1. 1.]",
+  "reasoning": "For state 0, the best action is to go to state 1 and get a reward of 1. For state 1, taking action 1 gives a reward of 1 and ends the episode, so its value is 1."
+}

questions/157_implement-the-bellman-equation-for-value-iteration/learn.md

@@ -0,0 +1,42 @@
+# **The Bellman Equation**
+
+The **Bellman equation** is a fundamental recursive equation in reinforcement learning that relates the value of a state to the values of possible next states. It provides the mathematical foundation for key RL algorithms such as value iteration and Q-learning.
+
+---
+
+## **Key Idea**
+For each state $s$, the value $V(s)$ is the maximum expected return obtainable by choosing the best action $a$ and then following the optimal policy:
+
+$$
+V(s) = \max_{a} \sum_{s'} P(s'|s, a) \left[ R(s, a, s') + \gamma V(s') \right]
+$$
+
+Where:
+- $V(s)$: value of state $s$
+- $a$: possible actions
+- $P(s'|s, a)$: probability of moving to state $s'$ from $s$ via $a$
+- $R(s, a, s')$: reward for this transition
+- $\gamma$: discount factor ($0 \leq \gamma \leq 1$)
+- $V(s')$: value of next state
+
+---
+
+## **How to Use**
+1. **For each state:**
+   - For each possible action, sum over possible next states, weighting by transition probability.
+   - Add the immediate reward and the discounted value of the next state.
+   - Choose the action with the highest expected value (for control).
+2. **Repeat until values converge** (value iteration) or as part of other RL updates.
+
+---
+
+## **Applications**
+- **Value Iteration** and **Policy Iteration** in Markov Decision Processes (MDP)
+- **Q-learning** and other RL algorithms
+- Calculating the optimal value function and policy in gridworlds, games, and general MDPs
+
+---
+
+## **Why It Matters**
+- The Bellman equation formalizes the notion of **optimality** in sequential decision-making.
+- It is a backbone for teaching agents to solve environments with rewards, uncertainty, and long-term planning.

questions/157_implement-the-bellman-equation-for-value-iteration/meta.json

@@ -0,0 +1,15 @@
+{
+  "id": "157",
+  "title": "Implement the Bellman Equation for Value Iteration",
+  "difficulty": "medium",
+  "category": "Reinforcement Learning",
+  "video": "",
+  "likes": "0",
+  "dislikes": "0",
+  "contributor": [
+    {
+      "profile_link": "https://github.com/moe18",
+      "name": "Moe Chabot"
+    }
+  ]
+}

questions/157_implement-the-bellman-equation-for-value-iteration/solution.py

@@ -0,0 +1,14 @@
+import numpy as np
+
+def bellman_update(V, transitions, gamma):
+    n_states = len(V)
+    new_V = np.zeros_like(V)
+    for s in range(n_states):
+        action_values = []
+        for a in transitions[s]:
+            total = 0
+            for prob, next_s, reward, done in transitions[s][a]:
+                total += prob * (reward + gamma * (0 if done else V[next_s]))
+            action_values.append(total)
+        new_V[s] = max(action_values)
+    return new_V

questions/157_implement-the-bellman-equation-for-value-iteration/starter_code.py

@@ -0,0 +1,14 @@
+import numpy as np
+
+def bellman_update(V, transitions, gamma):
+    """
+    Perform one step of value iteration using the Bellman equation.
+    Args:
+      V: np.ndarray, state values, shape (n_states,)
+      transitions: list of dicts. transitions[s][a] is a list of (prob, next_state, reward, done)
+      gamma: float, discount factor
+    Returns:
+      np.ndarray, updated state values
+    """
+    # TODO: Implement Bellman update
+    pass

questions/157_implement-the-bellman-equation-for-value-iteration/tests.json

@@ -0,0 +1,10 @@
+[
+  {
+    "test": "import numpy as np\ntransitions = [\n  # For state 0\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, False)]},\n  # For state 1\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, True)]}\n]\nV = np.array([0.0, 0.0])\ngamma = 0.9\nnew_V = bellman_update(V, transitions, gamma)\nprint(np.round(new_V, 2))",
+    "expected_output": "[1., 1.]"
+  },
+  {
+    "test": "import numpy as np\ntransitions = [\n  {0: [(0.8, 0, 5, False), (0.2, 1, 10, False)], 1: [(1.0, 1, 2, False)]},\n  {0: [(1.0, 0, 0, False)], 1: [(1.0, 1, 0, True)]}\n]\nV = np.array([0.0, 0.0])\ngamma = 0.5\nnew_V = bellman_update(V, transitions, gamma)\nprint(np.round(new_V, 2))",
+    "expected_output": "[6.,  0.]"
+  }
+]

utils/convert_single_question.py

@@ -28,130 +28,38 @@
 
 # ── 1️⃣  EDIT YOUR QUESTION HERE ────────────────────────────────────────────
 QUESTION_DICT: Dict[str, Any] = {
-    "id": "143",
-    "title": "Instance Normalization (IN) Implementation",
-    "description": "Implement the Instance Normalization operation for 4D tensors (B, C, H, W) using NumPy. For each instance in the batch and each channel, normalize the spatial dimensions (height and width) by subtracting the mean and dividing by the standard deviation, then apply a learned scale (gamma) and shift (beta).",
-    "test_cases": [
-        {
-            "test": "import numpy as np\nB, C, H, W = 2, 2, 2, 2\nnp.random.seed(42)\nX = np.random.randn(B, C, H, W)\ngamma = np.ones(C)\nbeta = np.zeros(C)\nout = instance_normalization(X, gamma, beta)\nprint(np.round(out[1][1], 4))",
-            "expected_output": "[[ 1.4005, -1.0503] [-0.8361, 0.486 ]]"
-        },
-        {
-            "test": "import numpy as np\nB, C, H, W = 2, 2, 2, 2\nnp.random.seed(101)\nX = np.random.randn(B, C, H, W)\ngamma = np.ones(C)\nbeta = np.zeros(C)\nout = instance_normalization(X, gamma, beta)\nprint(np.round(out[1][0], 4))",
-            "expected_output": "[[-1.537, 0.9811], [ 0.7882, -0.2323]]"
-        },
-        {
-            "test": "import numpy as np\nB, C, H, W = 2, 2, 2, 2\nnp.random.seed(101)\nX = np.random.randn(B, C, H, W)\ngamma = np.ones(C) * 0.5\nbeta = np.ones(C)\nout = instance_normalization(X, gamma, beta)\nprint(np.round(out[0][0], 4))",
-            "expected_output": "[[1.8542, 0.6861], [0.8434, 0.6163]]"
-        }
-    ],
-    "solution": "import numpy as np\n\ndef instance_normalization(X: np.ndarray, gamma: np.ndarray, beta: np.ndarray, epsilon: float = 1e-5) -> np.ndarray:\n    # Reshape gamma, beta for broadcasting: (1, C, 1, 1)\n    gamma = gamma.reshape(1, -1, 1, 1)\n    beta = beta.reshape(1, -1, 1, 1)\n    mean = np.mean(X, axis=(2, 3), keepdims=True)\n    var = np.var(X, axis=(2, 3), keepdims=True)\n    X_norm = (X - mean) / np.sqrt(var + epsilon)\n    return gamma * X_norm + beta",
-    "example": {
-        "input": "import numpy as np\nB, C, H, W = 2, 2, 2, 2\nnp.random.seed(42)\nX = np.random.randn(B, C, H, W)\ngamma = np.ones(C)\nbeta = np.zeros(C)\nout = instance_normalization(X, gamma, beta)\nprint(np.round(out, 8))",
-        "output": "[[[[-0.08841405 -0.50250083]\n   [ 0.01004046  0.58087442]]\n\n  [[-0.43833369 -0.43832346]\n   [ 0.69114093  0.18551622]]]\n\n [[[-0.17259136  0.51115219]\n   [-0.16849938 -0.17006144]]\n\n  [[ 0.73955155 -0.55463639]\n   [-0.44152783  0.25661268]]]]",
-        "reasoning": "The function normalizes each instance and channel across (H, W), then applies the gamma and beta scaling/shifting parameters. This matches standard InstanceNorm behavior."
+  "id": "157",
+  "title": "Implement the Bellman Equation for Value Iteration",
+  "description": "Write a function that performs one step of value iteration for a given Markov Decision Process (MDP) using the Bellman equation. The function should update the state-value function V(s) for each state based on possible actions, transition probabilities, rewards, and the discount factor gamma. Only use NumPy.",
+  "test_cases": [
+    {
+      "test": "import numpy as np\ntransitions = [\n  # For state 0\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, False)]},\n  # For state 1\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, True)]}\n]\nV = np.array([0.0, 0.0])\ngamma = 0.9\nnew_V = bellman_update(V, transitions, gamma)\nprint(np.round(new_V, 2))",
+      "expected_output": "[1., 1.]"
     },
-    "category": "Deep Learning",
-    "starter_code": "import numpy as np\n\ndef instance_normalization(X: np.ndarray, gamma: np.ndarray, beta: np.ndarray, epsilon: float = 1e-5) -> np.ndarray:\n    \"\"\"\n    Perform Instance Normalization over a 4D tensor X of shape (B, C, H, W).\n    gamma: scale parameter of shape (C,)\n    beta: shift parameter of shape (C,)\n    epsilon: small value for numerical stability\n    Returns: normalized array of same shape as X\n    \"\"\"\n    # TODO: Implement Instance Normalization\n    pass",
-    "learn_section": r"""## Understanding Instance Normalization
-
-Instance Normalization (IN) is a normalization technique primarily used in image generation and style transfer tasks. Unlike Batch Normalization or Group Normalization, Instance Normalization normalizes each individual sample (or instance) separately, across its spatial dimensions. This is particularly effective in applications like style transfer, where normalization is needed per image to preserve the content while allowing different styles to be applied.
-
-### Concepts
-
-Instance Normalization operates on the principle of normalizing each individual sample independently. This helps to remove the style information from the images, leaving only the content. By normalizing each instance, the method allows the model to focus on the content of the image rather than the variations between images in a batch.
-
-The process of Instance Normalization consists of the following steps:
-
-1. **Compute the Mean and Variance for Each Instance:** For each instance (image), compute the mean and variance across its spatial dimensions.
-2. **Normalize the Inputs:** Normalize each instance using the computed mean and variance.
-3. **Apply Scale and Shift:** After normalization, apply a learned scale (gamma) and shift (beta) to restore the model's ability to represent the data's original distribution.
-
-### Structure of Instance Normalization for BCHW Input
-
-For an input tensor with the shape **BCHW** , where:
-- **B**: batch size,
-- **C**: number of channels,
-- **H**: height,
-- **W**: width,
-Instance Normalization operates on the spatial dimensions (height and width) of each instance (image) separately.
-
-#### 1. Mean and Variance Calculation for Each Instance
-
-- For each individual instance in the batch (for each **b** in **B**), the **mean** $\mu_b$ and **variance** $\sigma_b^2$ are computed across the spatial dimensions (height and width), but **independently for each channel**.
-
-  $$ 
-  \mu_b = \frac{1}{H \cdot W} \sum_{h=1}^{H} \sum_{w=1}^{W} x_{b,c,h,w}
-  $$
-
-  $$
-  \sigma_b^2 = \frac{1}{H \cdot W} \sum_{h=1}^{H} \sum_{w=1}^{W} (x_{b,c,h,w} - \mu_b)^2
-  $$
-
-  Where:
-  - $x_{b,c,h,w}$ is the activation at batch index $b$, channel $c$, height $h$, and width $w$.
-  - $H$ and $W$ are the spatial dimensions (height and width).
-
-#### 2. Normalization
-
-Once the mean $\mu_b$ and variance $\sigma_b^2$ have been computed for each instance, the next step is to **normalize** the input for each instance across the spatial dimensions (height and width), for each channel:
-
-$$
-\hat{x}_{b,c,h,w} = \frac{x_{b,c,h,w} - \mu_b}{\sqrt{\sigma_b^2 + \epsilon}}
-$$
-
-Where:
-- $\hat{x}_{b,c,h,w}$ is the normalized activation for the input at batch index $b$, channel index $c$, height $h$, and width $w$.
-- $\epsilon$ is a small constant added to the variance for numerical stability.
-
-#### 3. Scale and Shift
-
-After normalization, the next step is to apply a **scale** ($\gamma_c$) and **shift** ($\beta_c$) to the normalized activations for each channel. These learned parameters allow the model to adjust the output distribution for each channel:
-
-$$
-y_{b,c,h,w} = \gamma_c \hat{x}_{b,c,h,w} + \beta_c
-$$
-
-Where:
-- $\gamma_c$ is the scaling factor for channel $c$.
-- $\beta_c$ is the shifting factor for channel $c$.
-
-#### 4. Training and Inference
-
-- **During Training**: The mean and variance are computed for each instance in the mini-batch and used for normalization.
-- **During Inference**: The model uses the running averages of the statistics (mean and variance) computed during training to ensure consistent behavior in production.
-
-### Key Points
-
-- **Instance-wise Normalization**: Instance Normalization normalizes each image independently, across its spatial dimensions (height and width) and across the channels.
-  
-- **Style Transfer**: This normalization technique is widely used in **style transfer** tasks, where each image must be normalized independently to allow for style information to be adjusted without affecting the content.
-
-- **Batch Independence**: Instance Normalization does not depend on the batch size, as normalization is applied per instance, making it suitable for tasks where per-image normalization is critical.
-
-- **Numerical Stability**: A small constant $\epsilon$ is added to the variance to avoid numerical instability when dividing by the square root of the variance.
-
-- **Improved Training in Style-Related Tasks**: Instance Normalization helps to remove unwanted style-related variability across different images, allowing for better performance in tasks like style transfer, where the goal is to separate content and style information.
-
-### Why Normalize Over Instances?
-
-- **Content Preservation**: By normalizing each image individually, Instance Normalization allows the model to preserve the content of the images while adjusting the style. This makes it ideal for style transfer and other image manipulation tasks.
-  
-- **Batch Independence**: Unlike Batch Normalization, which requires large batch sizes to compute statistics, Instance Normalization normalizes each image independently, making it suitable for tasks where the batch size is small or varies.
-
-- **Reducing Style Variability**: Instance Normalization removes the variability in style information across a batch, allowing for a consistent representation of content across different images.
-
-In summary, Instance Normalization is effective for image-based tasks like style transfer, where the goal is to normalize each image independently to preserve its content while allowing style modifications.""",
-    "contributor": [
-        {
-            "profile_link": "https://github.com/nzomi",
-            "name": "nzomi"
-        }
-    ],
-    "likes": "0",
-    "dislikes": "0",
-    "difficulty": "medium",
-    "video": ""
+    {
+      "test": "import numpy as np\ntransitions = [\n  {0: [(0.8, 0, 5, False), (0.2, 1, 10, False)], 1: [(1.0, 1, 2, False)]},\n  {0: [(1.0, 0, 0, False)], 1: [(1.0, 1, 0, True)]}\n]\nV = np.array([0.0, 0.0])\ngamma = 0.5\nnew_V = bellman_update(V, transitions, gamma)\nprint(np.round(new_V, 2))",
+      "expected_output": "[6.,  0.]"
+    }
+  ],
+  "solution": "import numpy as np\n\ndef bellman_update(V, transitions, gamma):\n    n_states = len(V)\n    new_V = np.zeros_like(V)\n    for s in range(n_states):\n        action_values = []\n        for a in transitions[s]:\n            total = 0\n            for prob, next_s, reward, done in transitions[s][a]:\n                total += prob * (reward + gamma * (0 if done else V[next_s]))\n            action_values.append(total)\n        new_V[s] = max(action_values)\n    return new_V",
+  "example": {
+    "input": "import numpy as np\ntransitions = [\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, False)]},\n  {0: [(1.0, 0, 0.0, False)], 1: [(1.0, 1, 1.0, True)]}\n]\nV = np.array([0.0, 0.0])\ngamma = 0.9\nnew_V = bellman_update(V, transitions, gamma)\nprint(np.round(new_V, 2))",
+    "output": "[1. 1.]",
+    "reasoning": "For state 0, the best action is to go to state 1 and get a reward of 1. For state 1, taking action 1 gives a reward of 1 and ends the episode, so its value is 1."
+  },
+  "category": "Reinforcement Learning",
+  "starter_code": "import numpy as np\n\ndef bellman_update(V, transitions, gamma):\n    \"\"\"\n    Perform one step of value iteration using the Bellman equation.\n    Args:\n      V: np.ndarray, state values, shape (n_states,)\n      transitions: list of dicts. transitions[s][a] is a list of (prob, next_state, reward, done)\n      gamma: float, discount factor\n    Returns:\n      np.ndarray, updated state values\n    \"\"\"\n    # TODO: Implement Bellman update\n    pass",
+  "learn_section": "# **The Bellman Equation**\n\nThe **Bellman equation** is a fundamental recursive equation in reinforcement learning that relates the value of a state to the values of possible next states. It provides the mathematical foundation for key RL algorithms such as value iteration and Q-learning.\n\n---\n\n## **Key Idea**\nFor each state $s$, the value $V(s)$ is the maximum expected return obtainable by choosing the best action $a$ and then following the optimal policy:\n\n$$\nV(s) = \\max_{a} \\sum_{s'} P(s'|s, a) \\left[ R(s, a, s') + \\gamma V(s') \\right]\n$$\n\nWhere:\n- $V(s)$: value of state $s$\n- $a$: possible actions\n- $P(s'|s, a)$: probability of moving to state $s'$ from $s$ via $a$\n- $R(s, a, s')$: reward for this transition\n- $\\gamma$: discount factor ($0 \\leq \\gamma \\leq 1$)\n- $V(s')$: value of next state\n\n---\n\n## **How to Use**\n1. **For each state:**\n   - For each possible action, sum over possible next states, weighting by transition probability.\n   - Add the immediate reward and the discounted value of the next state.\n   - Choose the action with the highest expected value (for control).\n2. **Repeat until values converge** (value iteration) or as part of other RL updates.\n\n---\n\n## **Applications**\n- **Value Iteration** and **Policy Iteration** in Markov Decision Processes (MDP)\n- **Q-learning** and other RL algorithms\n- Calculating the optimal value function and policy in gridworlds, games, and general MDPs\n\n---\n\n## **Why It Matters**\n- The Bellman equation formalizes the notion of **optimality** in sequential decision-making.\n- It is a backbone for teaching agents to solve environments with rewards, uncertainty, and long-term planning.",
+  "contributor": [
+    {
+      "profile_link": "https://github.com/moe18",
+      "name": "Moe Chabot"
+    }
+  ],
+  "likes": "0",
+  "dislikes": "0",
+  "difficulty": "medium",
+  "video": ""
 }