PHP cURL not returning expected HTML content with redirects: How to match command-line curl results?

General
1

I’m trying to run a curl query to my website to retreive the body, it should be a html returned. There is a redirect before reaching the final url. When I run it from my command line, I get the expected result :

I’m attempting to execute a curl command on my website to retrieve the HTML content of the page. There’s a redirect that occurs before reaching the final URL. When I run the command in my terminal, the result is as expected.

curl.exe https://example.com/saw/ess?TENANTID=123-L

Response:

<!DOCTYPE HTML>
<html>
<head>
    <title>Bad request</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <meta http-equiv="x-ua-compatible" content="IE=edge" />
    <link rel="shortcut icon" href="/bo/favicon.ico"/>
    <style>
        @font-face {
            font-family: 'Metric-Light';
            src: url('/bo/fonts/Metric-Light.eot?#iefix') format('embedded-opentype'),
            url('/bo/fonts/Metric-Light.woff') format('woff'),
            url('/bo/fonts/Metric-Light.ttf') format('truetype')
        }

        body {
            font-family: "Metric-Light", "Helvetica Neue", "Helvetica", "Arial", "sans-serif";
        }

        .content {

        }

        .logo {
            display: block;
            margin-top: 10%;
            margin-left: auto;
            margin-right: auto;
        }

        .title{
            font-size: 70px;
            text-align: center;
            color: #6f6f6f;
            text-transform: capitalize;
            display: block;
            margin-top: 3%;
        }

        .exception{
            font-size: 30px;
            text-align: center;
            color: #6f6f6f;
            display: block;
            margin-top: 15px;
        }

        .detail{
            margin-top: 20px;
            font-size: 20px;
            text-align: center;
            color: #6f6f6f;
            display: block;
        }

    </style>
</head>
<body>
<div class="content">
    <img class="logo" src="/bo/fonts/Bad_Request_Content.png" alt="error"/><br/>
    <span class="title">Bad request</span><br/>
    <span class="exception">Bad request</span>
    <p class="detail">Please try again or contact your local IT person.</p>
</div>
</body>
</html>

The result is as expected. Now, I’d like to retrieve the content using PHP.

I attempted the following code:

<?php
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "https://example.com/saw/ess?TENANTID=123");
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true); // Follow redirects
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// Set headers to mimic the Chrome curl command
curl_setopt($curl, CURLOPT_HTTPHEADER, [
    "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
]);

$response = curl_exec($curl);
if ($response === false) {
    echo "cURL Error: " . curl_error($curl);
    curl_close($curl);
    exit;
}
curl_close($curl);
var_dump($response);

Output:

<!doctype html>
<html dir="ltr" lang="en">
<head>
  <meta charset="utf-8">
  <title>Idm Service</title>
  <base href="/idm-service/idm/v0/">
  <meta name="viewport" http-equiv="X-UA-Compatible" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
<link rel="stylesheet" href="styles.a06961a3935e65df.css"></head>
<body>
  <app-root></app-root>
<script src="runtime.36808ec0bfe1e098.js" type="module"></script><script src="polyfills.abbcd89f1033283d.js" type="module"></script><script src="main.99cb60abd387e4e1.js" type="module"></script></body>
</html>

How can I retrieve the same output as in the command line?

2

The difference in output between your curl command line and the PHP code likely stems from how they handle redirects differently. Here’s how to make your PHP code retrieve the same HTML content as the command line:

1. Verify Redirect Handling:

  • Check if the command line tool you’re using (e.g., curl.exe on Windows) has automatic redirect following enabled by default.
  • Ensure your PHP code explicitly follows redirects using CURLOPT_FOLLOWLOCATION :

PHP

$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "https://example.com/saw/ess?TENANTID=123");
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);  // Follow redirects (already set)
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);  // Set headers to mimic the Chrome curl command
// ... other options ...

2. Consider Additional Options:

  • HTTP Headers: Setting a user-agent may not be necessary if the server doesn’t perform user-agent-based filtering.
  • Error Handling: Ensure your code checks for curl errors with curl_errno($curl) .
  • Verbose Output (Optional): Add curl_setopt($curl, CURLOPT_VERBOSE, true); to see detailed output during the request.

Revised Code:

PHP

<?php

$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "https://example.com/saw/ess?TENANTID=123");
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);  // Follow redirects
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);  // Return content

// Optional: Check for curl errors
if (curl_errno($curl)) {
  echo "cURL Error: " . curl_error($curl);
  curl_close($curl);
  exit;
}

$response = curl_exec($curl);
curl_close($curl);

if ($response === false) {
  echo "An error occurred while fetching the content.";
  exit;
}

var_dump($response);

Explanation:

  • The code keeps the essential options for following redirects and returning content.
  • It includes a basic check for curl errors using curl_errno($curl) .
  • Comments explain the purpose of each option.

Additional Tips:

  • Customize error handling based on your specific needs.
  • For more complex scenarios, consider the curl_getinfo() function to retrieve additional information about the request.

By incorporating these suggestions, your PHP code should retrieve the same HTML content as the command line, even in the presence of redirects.