Update the data processing to remove missing values

[svpino]

The training pipeline should remove any missing values right after loading the dataset.

[traycerai]

Plan

Observations

• The dataset is loaded in load_dataset() (in pipelines/common.py), which currently reads the CSV, replaces extraneous sex values with NaN, shuffles, and returns.
• No rows are dropped, so missing values persist until feature‐engineer imputers run downstream.
• The ticket requests immediate removal of missing values right after loading, and existing tests (tests/test_common_flowmixin.py) assert original lengths and will break if rows are dropped.

Approach

• In load_dataset(), after replacing sex anomalies with NaN, insert a dropna() call to remove any rows containing missing values. Log how many rows were dropped for observability.
• Ensure dropna() executes before the shuffle step so all downstream components see only complete records.
• Update the sex‐column cleaning docstring to indicate that missing‐value removal occurs.
• Adjust the test_load_dataset in tests/test_common_flowmixin.py to assert that no missing values remain instead of comparing lengths.

Reasoning

• Inspected pipelines/training.py and traced data loading to DatasetMixin.load_dataset() in pipelines/common.py.
• Reviewed tests in tests/test_common_flowmixin.py to see current expectations on row counts and sex cleaning.
• Confirmed that removing missing values at load time aligns with the ticket requirement and identified impacted tests.

Proposed File Changes

📄 pipelines/common.py ^(MODIFY) 🔗

Referred Files

• 📄 pipelines/common.py ^(MODIFY)

• In the load_dataset() method of pipelines/common.py, immediately after the line that replaces extraneous sex values with NaN (around line 44), insert:

  row_count_before = len(data)
  data = data.dropna()
  logging.info("Dropped %d rows with missing values", row_count_before - len(data))

• Ensure this comes before the shuffle (data.sample(...)) step.
• Update the docstring of load_dataset() to note that missing rows are dropped.

📄 tests/test_common_flowmixin.py ^(MODIFY) 🔗

Referred Files

• 📄 tests/test_common_flowmixin.py ^(MODIFY)

• Replace the existing test_load_dataset in tests/test_common_flowmixin.py so it no longer asserts original length. Instead, assert that the returned dataset contains no missing values. For example:

  def test_load_dataset(metaflow_data):  
      assert not metaflow_data.data.isna().any().any(), "Dataset should have no missing values"

• Remove or comment out the old length‐equality assertion.

Diagram

sequenceDiagram
    participant StartStep
    participant Loader as load_dataset()
    participant DataFrame
    StartStep->>Loader: call load_dataset()
    Loader->>DataFrame: read CSV & replace sex "." with NaN
    Loader->>DataFrame: drop rows with missing values
    Loader->>Loader: shuffle rows
    Loader-->>StartStep: return cleaned DataFrame

Loading

Import In IDE

Developer Humor

“I dropped all the missing values—now our data has commitment issues!” 😂
(Rows: 0, NaNs: 0)🚀

---

Execution Information

Branch: ai
Commit: c1c3cdb

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.