Description
I'm trying to create a bridge, I'm a beginner.
When you first come to the webpage I'm trying to RSS-bridge you have a form where you select multiple inputs but it already show you the firsts results without your inputs.
I successfully managed to create the bridge on this page without the form inputs configured with getSimpleHTMLDOM.
Now I'm trying to go further by making it work with the desired inputs.
The form has multiple inputs fields and a submit button that seems to launch some javascript, then you came on the same page with the results customised.
I first tried to pass the POST parameters to the getSimpleHTMLDOM url, but it seems it is only affecting the form and not the results.
I found that cookies are created with the form inputs. I guess it is by this way that the inputs are transmitted?
There is a PHPSESSION cookie that is created too. Can it be used to transmit the inputs too?
I tried this but I think it's not the right way: (I write the desiredvalues harcoded for my tests.)
$arr_cookie_options = array (
'expires' => time() + 60*60*24*30,
'path' => '/',
'domain' => 'www.websiteurl.com', // leading dot for compatibility or use subdomain
'secure' => false, // or false
'httponly' => false, // or false
'samesite' => 'None' // None || Lax || Strict
);
setcookie('ad_dep', 'desiredvalue', $arr_cookie_options);
setcookie('av_arr', 'desiredvalue', $arr_cookie_options);
setcookie('av_dep', 'desiredvalue', $arr_cookie_options);
setcookie('date_dep', 'desiredvalue', $arr_cookie_options);
setcookie('h_deb_r', 'desiredvalue', $arr_cookie_options);
setcookie('h_fin_r', 'desiredvalue', $arr_cookie_options);
setcookie('PHPSESSID', 'lastidfromfirefox', $arr_cookie_options);
setcookie('places_r', 'desiredvalue', $arr_cookie_options);
setcookie('type_li', 'desiredvalue', $arr_cookie_options);
setcookie('ville_arr', 'desiredvalue', $arr_cookie_options);
setcookie('ville_dep', 'desiredvalue', $arr_cookie_options);
$domdelapage = getSimpleHTMLDOM('https://www.websiteurl.com/search.html?L_cal_date=&L_jour_dep=desiredvalue&L_mois_annee_dep=desiredvalue&ville_dep=desiredvalue&L_places_r=desiredvalue&ville_arr=desiredvalue&L_h_deb_r=desiredvalue&L_h_fin_r=desiredvalue&rech_autour=O&L_lebouton=&L_nbre_enreg=-1&L_javascript=O');
I guess it is not the proper way to set the cookies. I should maybe follow the PepperBridge ?
With something like that:
// Get Cookies header to do the query
$cookiesHeaderValue = $this->getCookiesHeaderValue($url);
// CURL Options
$opts = [
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $queryJSON
];
$json = getContents($url, $header, $opts);
I guess it would be better to get the PHPSESSID from the website. I should maybe follow the PixivBridge?
With something like that:
private function checkCookie(array $headers)
{
if (array_key_exists('set-cookie', $headers))
{
foreach ($headers['set-cookie'] as $value)
{
if (str_starts_with($value, 'PHPSESSID='))
{
parse_str(strtr($value, ['&' => '%26', '+' => '%2B', ';' => '&']), $cookie);
if ($cookie['PHPSESSID'] != $this->getCookie())
{
$this->saveCacheValue('cookie', $cookie['PHPSESSID']);
}
break;
}
}
}
}
private function getCookie()
{
// checks if cookie is set, if not initialise it with the cookie from the config
$value = $this->loadCacheValue('cookie');
if (!$value)
{
$value = $this->getOption('cookie');
// 30 days + 1 day to let cookie chance to renew
$this->saveCacheValue('cookie', $this->getOption('cookie'), 2678400);
}
return $value;
}
//Cache getContents by default
private function getData(string $url, bool $cache = true, bool $getJSON = false, array $httpHeaders = [], array $curlOptions = [])
{
$cookie_str = $this->getCookie();
if ($cookie_str)
{
$curlOptions[CURLOPT_COOKIE] = 'PHPSESSID=' . $cookie_str;
}
if ($cache)
{
$response = $this->loadCacheValue($url);
if (!$response || is_array($response))
{
$response = getContents($url, $httpHeaders, $curlOptions, true);
$this->saveCacheValue($url, $response);
}
}
else
{
$response = getContents($url, $httpHeaders, $curlOptions, true);
}
$this->checkCookie($response->getHeaders());
if ($getJSON)
{
return json_decode($response->getBody(), true);
}
return $response->getBody();
}
On a shell with curl I get the good customized results with this:
curl 'https://www.websiteurl.com/search.html' --compressed -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.websiteurl.com' -H 'Connection: keep-alive' -H 'Referer: https://www.websiteurl.com/search.html' -H 'Cookie: PHPSESSID=lastidfromfirefox; ville_dep=desiredvalue; ville_arr=desiredvalue; date_dep=desiredvalue; places_r=desiredvalue; h_deb_r=desiredvalue; h_fin_r=desiredvalue; type_li=desiredvalue; av_dep=desiredvalue; av_arr=desiredvalue; ad_dep=desiredvalue' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-User: ?1' -H 'Priority: u=0, i' --data-raw 'L_cal_date=&L_jour_dep=desiredvalue&L_mois_annee_dep=desiredvalue&ville_dep=desiredvalue&L_places_r=desiredvalue&ville_arr=desiredvalue&L_h_deb_r=desiredvalue&L_h_fin_r=desiredvalue&rech_autour=O&L_lebouton=&L_nbre_enreg=-1&L_javascript=O'
What the best way to follow?
Thank you for your help (and sorry for my bad English)