fix: address selection in Broxbourne Council#1873
fix: address selection in Broxbourne Council#1873teofanis wants to merge 1 commit intorobbrad:masterfrom
Conversation
📝 WalkthroughWalkthroughUpdates the BroxbourneCouncil web scraping module to use a more specific XPath for address selection and adds explicit clearing of the postcode input field before entering a new value, improving robustness of the address lookup flow. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1873 +/- ##
=======================================
Coverage 86.67% 86.67%
=======================================
Files 9 9
Lines 1141 1141
=======================================
Hits 989 989
Misses 152 152 ☔ View full report in Codecov by Sentry. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py`:
- Around line 68-76: The code waits only for the <select> element before calling
Select(...).select_by_value(user_uprn), which can fail if options load
asynchronously; update BroxbourneCouncil.py to first wait for the option with
the desired UPRN to appear (e.g., use WebDriverWait +
EC.presence_of_element_located with an XPath like
"//label[normalize-space()='Choose
address']/following::select[1]/option[`@value`='{user_uprn}']"), then call
Select(address_select).select_by_value(user_uprn) so selection only occurs after
the option is present.
| address_select = WebDriverWait(driver, 15).until( | ||
| EC.presence_of_element_located((By.XPATH, "//select")) | ||
| EC.presence_of_element_located( | ||
| ( | ||
| By.XPATH, | ||
| "//label[normalize-space()='Choose address']/following::select[1]", | ||
| ) | ||
| ) | ||
| ) | ||
| Select(address_select).select_by_value(user_uprn) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "BroxbourneCouncil.py" | head -5Repository: robbrad/UKBinCollectionData
Length of output: 138
🏁 Script executed:
git ls-files | grep -i broxbourneRepository: robbrad/UKBinCollectionData
Length of output: 136
🏁 Script executed:
sed -n '60,85p' uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.pyRepository: robbrad/UKBinCollectionData
Length of output: 970
🏁 Script executed:
sed -n '40,100p' uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.pyRepository: robbrad/UKBinCollectionData
Length of output: 2411
🏁 Script executed:
grep -n "try\|except\|select_by_value" uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py | head -20Repository: robbrad/UKBinCollectionData
Length of output: 317
🏁 Script executed:
grep -r "select_by_value" uk_bin_collection/uk_bin_collection/councils/ | head -10Repository: robbrad/UKBinCollectionData
Length of output: 1373
🏁 Script executed:
grep -B5 -A5 "select_by_value" uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.pyRepository: robbrad/UKBinCollectionData
Length of output: 520
🏁 Script executed:
grep -n "Cannot locate option" uk_bin_collection/uk_bin_collection/councils/*.pyRepository: robbrad/UKBinCollectionData
Length of output: 53
🏁 Script executed:
grep -B10 "select_by_value" uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py | grep -E "try|except|wait"Repository: robbrad/UKBinCollectionData
Length of output: 53
🏁 Script executed:
sed -n '1,30p' uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.pyRepository: robbrad/UKBinCollectionData
Length of output: 1274
🏁 Script executed:
grep -n "import\|from" uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py | head -15Repository: robbrad/UKBinCollectionData
Length of output: 525
🏁 Script executed:
sed -n '28,120p' uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py | tail -30Repository: robbrad/UKBinCollectionData
Length of output: 1529
🏁 Script executed:
python3 << 'EOF'
# Verify the proposed fix syntax would work with Selenium
# Check if lambda with Select.options would work
from selenium.webdriver.support.ui import Select
# The proposed fix uses:
# WebDriverWait(driver, 15).until(
# lambda d: any(
# opt.get_attribute("value") == user_uprn
# for opt in Select(address_select).options
# )
# )
# This syntax is correct for Selenium - Select.options returns list of WebElements
# and get_attribute is a valid WebElement method
print("Proposed fix syntax is valid for Selenium")
print("Lambda will check if any option's value matches user_uprn")
print("This prevents 'Cannot locate option' errors by ensuring option exists before select_by_value")
EOFRepository: robbrad/UKBinCollectionData
Length of output: 191
Wait for the UPRN option to exist before calling select_by_value.
At line 76, select_by_value is called immediately after waiting only for the <select> element's presence in the DOM. This doesn't guarantee that options are populated if they load asynchronously. Add an explicit wait for the option value to appear in the dropdown before attempting selection to prevent "Cannot locate option with value ..." failures.
Proposed fix
# Select address
address_select = WebDriverWait(driver, 15).until(
EC.presence_of_element_located(
(
By.XPATH,
"//label[normalize-space()='Choose address']/following::select[1]",
)
)
)
+ WebDriverWait(driver, 15).until(
+ lambda d: any(
+ opt.get_attribute("value") == user_uprn
+ for opt in Select(address_select).options
+ )
+ )
Select(address_select).select_by_value(user_uprn)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py` around
lines 68 - 76, The code waits only for the <select> element before calling
Select(...).select_by_value(user_uprn), which can fail if options load
asynchronously; update BroxbourneCouncil.py to first wait for the option with
the desired UPRN to appear (e.g., use WebDriverWait +
EC.presence_of_element_located with an XPath like
"//label[normalize-space()='Choose
address']/following::select[1]/option[`@value`='{user_uprn}']"), then call
Select(address_select).select_by_value(user_uprn) so selection only occurs after
the option is present.
1 participant
The addition
<select>elements shown above leak into our lookup and causeCannot locate option with value ...errors when we try to select the UPRN. This change makes the selector target theChoose addressdrop-down explicitly.Summary by CodeRabbit