Gamedev A/B Test Analysis: Difference-in-Differences (DiD)

This repository contains an analysis of a game experiment involving changes to player settings and configurations. The primary goal is to evaluate the impact of these changes on player combat performance using the Difference-in-Differences (DiD) methodology.

📋 Experiment Overview

The experiment was conducted between October 11, 2022, and November 29, 2022.

• Group A (Treatment): Received configuration changes starting October 27, 2022.
• Group B (Control): No changes were applied.
• Post-treatment Period: October 27, 2022 — November 29, 2022.

📊 Data Description

The data.csv file contains detailed battle logs:

• player_id, battle_id — Unique identifiers for players and battles.
• dt — Battle date.
• player_group — Group assignment (A or B).
• damage_dealt — Total damage dealt in battle.
• kills_made — Number of opponents destroyed (frags).
• in_battle_presence_time — Time spent in battle.
• vehicle_lvl — Vehicle tier/level.
• account_created_at — Player account registration date.

🛠 Methodology

To estimate the treatment effect, the Difference-in-Differences approach was used. This method helps isolate the effect of the update by accounting for time-based trends common to both groups. Key performance indicators (KPIs) analyzed:

1. Damage Dealt
2. Kills Made
3. DPM (Damage Per Minute)

🚀 Key Findings

Based on the analysis in DID_Gamedev.ipynb:

• No Significant Effect: DiD estimates across all key KPIs (damage, kills, DPM) are close to zero.
• Parallel Trends: Pre-release and post-release trends remain parallel for both groups, suggesting that observed fluctuations are driven by global temporal effects rather than the configuration update.
• Statistical Significance: No statistically meaningful improvement in player performance was observed in the treatment group.

1. Do Not Scale: The current version of the feature does not produce a measurable performance uplift.
2. Redesign Experiment: Consider re-running the experiment with clearer exposure tracking and randomization at the user level.
3. Expand Metrics: Future analysis should include additional outcome metrics such as Retention or Win Rate for a more comprehensive evaluation.