Move is_in_clearance to dynamic data

[simark]

I think we should move the "is_in_clearance" flag to the dynamic data. Right now, if a product not in clearance becomes in clearance, it won't be reflected in the database. Having it in the dynamic data table means we could see when it started being in clearance, along with how the price changed at the same moment.

[simark]

I tried to do this and realized why it's in the static data currently. There is always a "clearance" in the data returned in the product search (used for scrape-inventory). But in the price query, there is only a StoreClearance field if you include a store id:

$ curl 'https://api-triangle.canadiantire.ca/esb/PriceAvailability?Product=0730094P&Banner=CTR&Language=E&Store=0064' -A 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 RuxitSynthetic/1.0 v3227947402484948568 t4157550440124640339'
[{"Banner":"CTR","Store":"0066","SKU":"0730094","CheckDigit":"2","Product":"0730094P","Price":32.99,"Quantity":0,"Description":"CCM ASCENT YTH P","Messages":{"Warranty":"This product carries a 1 year exchange warranty redeemable at any Canadian Tire store."},"PartNumber":"RF60-Y","IsOnline":{"Active":"Y","Exclusive":"N","Sellable":"Y","Orderable":"N","StoreClearance":"N","SpecialBuy":"N"},"Corporate":{"Quantity":0,"MinOrderQty":4,"MinETA":7,"MaxETA":13},"PriceFrom":"N"}]

vs

curl 'https://api-triangle.canadiantire.ca/esb/PriceAvailability?Product=0730094P&Banner=CTR&Language=E' -A 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 RuxitSynthetic/1.0 v3227947402484948568 t4157550440124640339' 
[{"Banner":"CTR","Product":"0730094P","PrimarySKU":"0730094","CheckDigit":"2","SKU":"0730094","SKU_Count":1,"Price":32.99,"PriceFrom":"N","Description":"CCM ASCENT YTH P","Messages":{"Warranty":"This product carries a 1 year exchange warranty redeemable at any Canadian Tire store."},"PartNumber":"RF60-Y","IsOnline":{"Active":"Y","Exclusive":"N","Sellable":"Y","SpecialBuy":"N"}}]%   

The StoreClearance flag is probably store-specific.

[jgalar]

My original idea was to update is_in_clearance when we see the clearance attribute change. My assumption is that discontinued products become in clearance before they become permanently out of stock. In that sense, I wouldn't expect them to come back. But indeed, we wouldn't see when those products changed state.

So, I agree that it doesn't make much sense to keep it as part of the static data.

If my assumption is true, we could simply save the sampling date when a product is spotted as being in clearance. Otherwise, it can be sampled as another (separate) dynamic attribute. What do you think?

I feel tracking store-specific attributes is a bit out of scope for the moment (although it's something I want to do eventually).

[simark]

My original idea was to update is_in_clearance when we see the clearance attribute change. My assumption is that discontinued products become in clearance before they become permanently out of stock. In that sense, I wouldn't expect them to come back. But indeed, we wouldn't see when those products changed state.

So, I agree that it doesn't make much sense to keep it as part of the static data.

If my assumption is true, we could simply save the sampling date when a product is spotted as being in clearance. Otherwise, it can be sampled as another (separate) dynamic attribute. What do you think?

Yeah, I think we can have a column in the "static" table for when the product went from !in_clearance to in_clearance. That's the most straightforward, since the in_clearance property comes from the product listing requests (which populate the static table).

I feel tracking store-specific attributes is a bit out of scope for the moment (although it's something I want to do eventually).

Agreed.