PHP 8.5 | Tokenizer/PHP: properly fix "Using null as an array offset" deprecation (#1230)

If a slash/hash comment is on its own line, the new line token is merged into the comment token for PHP 8.0+, and the new line is skipped by setting it to `null`.

The PHP Tokenizer did not take the possibility of a token being set to `null` into account when retokening a `?` to either `T_NULLABLE` or `T_INLINE_THEN`, leading to the PHP 8.5 deprecation notice.
In that case, the sniff first looks forward to see if we can draw a conclusion based on the non-empty tokens following the `?` and if not, walks backward to look at the tokens before the `?`.
The problem occurs when a `null` token exists in the sequence before the `?`.

This `null` token will only have been injected with a specific code sequence: when there is a slash/hash comment followed by a new line in the token sequence before the `?` and there is no indentation/new line whitespace on the next line.

Also see the detailed analysis by andrewnicols in PHPCSStandards/PHP_CodeSniffer 1216#issuecomment-3274198443

There are only two situations in which this causes the tokenizer to hit the PHP 8.5 "Using null as an array offset" deprecation notice:
1. If the `?` token is the last token in the file (live coding - the issue was discovered via a test related to live coding).
2. If there are tokens after the `?`, but not tokens which allows us to draw a conclusion (yet) and there is a slash/hash comment between the `?` and the previous non-empty token.

This commit:
1. Adds dedicated tests for both situations.
2. Adds a new fix for situation [1] as if there are no tokens after a `?`, we cannot determine definitively whether the `?` is a nullable operator or an inline then. For BC, this should be tokenized as `T_INLINE_THEN`.
3. Makes a small tweak to the previously added fix which addressed situation [2].
    Alternatively, we could consider switching to using the "$finalTokens" token stack to do the token walking instead, but as that is a bigger change and this part of the code has nowhere near sufficient tests, that change is left for later.

Includes reverting the error silencing which was put in place temporarily to allow for the issue to be investigated.

Fixes 1216

Ref: https://wiki.php.net/rfc/deprecations_php_8_5#deprecate_using_values_null_as_an_array_offset_and_when_calling_array_key_exists

---------

Co-authored-by: jrfnl <jrfnl@users.noreply.github.com>

Author: andrewnicols

src/Tokenizers/PHP.php

@@ -2240,6 +2240,24 @@ protected function tokenize($string)
                     break;
                 }//end for
 
+                // Handle live coding/parse errors elegantly.
+                // If the "?" is the last non-empty token in the file, we cannot draw a definitive conclusion,
+                // so tokenize as T_INLINE_THEN.
+                if ($i === $numTokens) {
+                    if (PHP_CODESNIFFER_VERBOSITY > 1) {
+                        echo "\t\t* token $stackPtr at end of file changed from ? to T_INLINE_THEN".PHP_EOL;
+                    }
+
+                    $newToken['code'] = T_INLINE_THEN;
+                    $newToken['type'] = 'T_INLINE_THEN';
+
+                    $insideInlineIf[] = $stackPtr;
+
+                    $finalTokens[$newStackPtr] = $newToken;
+                    $newStackPtr++;
+                    continue;
+                }
+
                 /*
                  * This can still be a nullable type or a ternary.
                  * Do additional checking.
@@ -2249,6 +2267,11 @@ protected function tokenize($string)
                 $lastSeenNonEmpty = null;
 
                 for ($i = ($stackPtr - 1); $i >= 0; $i--) {
+                    if (isset($tokens[$i]) === false) {
+                        // Ignore skipped tokens (related to PHP 8+ slash/hash comment vs new line retokenization).
+                        continue;
+                    }
+
                     if (is_array($tokens[$i]) === true) {
                         $tokenType = $tokens[$i][0];
                     } else {
@@ -2264,7 +2287,7 @@ protected function tokenize($string)
                     }
 
                     if ($prevNonEmpty === null
-                        && @isset(Tokens::$emptyTokens[$tokenType]) === false
+                        && isset(Tokens::$emptyTokens[$tokenType]) === false
                     ) {
                         // Found the previous non-empty token.
                         if ($tokenType === ':' || $tokenType === ',' || $tokenType === T_ATTRIBUTE_END) {
@@ -2283,8 +2306,8 @@ protected function tokenize($string)
 
                     if ($tokenType === T_FUNCTION
                         || $tokenType === T_FN
-                        || @isset(Tokens::$methodPrefixes[$tokenType]) === true
-                        || @isset(Tokens::$scopeModifiers[$tokenType]) === true
+                        || isset(Tokens::$methodPrefixes[$tokenType]) === true
+                        || isset(Tokens::$scopeModifiers[$tokenType]) === true
                         || $tokenType === T_VAR
                         || $tokenType === T_READONLY
                     ) {
@@ -2307,7 +2330,7 @@ protected function tokenize($string)
                         break;
                     }
 
-                    if (@isset(Tokens::$emptyTokens[$tokenType]) === false) {
+                    if (isset(Tokens::$emptyTokens[$tokenType]) === false) {
                         $lastSeenNonEmpty = $tokenType;
                     }
                 }//end for

tests/Core/Tokenizers/PHP/NullableVsInlineThenParseErrorTest.inc

@@ -0,0 +1,9 @@
+<?php
+
+// This should be the only test in the file.
+// Ref: https://github.com/PHPCSStandards/PHP_CodeSniffer/issues/1216
+
+/* testLiveCoding */
+$ternary = true
+    # This must be a slash or hash comment and the next line must **NOT** have any indentation for the PHP 8.5 deprecation notice to occur.
+? //comment.

tests/Core/Tokenizers/PHP/NullableVsInlineThenParseErrorTest.php

@@ -0,0 +1,40 @@
+<?php
+/**
+ * Tests the retokenization of ? to T_NULLABLE or T_INLINE_THEN.
+ *
+ * @copyright 2025 PHPCSStandards and contributors
+ * @license   https://github.com/PHPCSStandards/PHP_CodeSniffer/blob/master/licence.txt BSD Licence
+ */
+
+namespace PHP_CodeSniffer\Tests\Core\Tokenizers\PHP;
+
+use PHP_CodeSniffer\Tests\Core\Tokenizers\AbstractTokenizerTestCase;
+
+/**
+ * Tests the retokenization of ? to T_NULLABLE or T_INLINE_THEN.
+ *
+ * @covers PHP_CodeSniffer\Tokenizers\PHP::tokenize
+ */
+final class NullableVsInlineThenParseErrorTest extends AbstractTokenizerTestCase
+{
+
+
+    /**
+     * Verify that a "?" as the last functional token in a file (live coding) is tokenized as `T_INLINE_THEN`
+     * as it cannot yet be determined what the token would be once the code is finalized.
+     *
+     * @return void
+     */
+    public function testInlineThenAtEndOfFile()
+    {
+        $tokens     = $this->phpcsFile->getTokens();
+        $target     = $this->getTargetToken('/* testLiveCoding */', [T_NULLABLE, T_INLINE_THEN]);
+        $tokenArray = $tokens[$target];
+
+        $this->assertSame(T_INLINE_THEN, $tokenArray['code'], 'Token tokenized as '.$tokenArray['type'].', not T_INLINE_THEN (code)');
+        $this->assertSame('T_INLINE_THEN', $tokenArray['type'], 'Token tokenized as '.$tokenArray['type'].', not T_INLINE_THEN (type)');
+
+    }//end testInlineThenAtEndOfFile()
+
+
+}//end class

tests/Core/Tokenizers/PHP/NullableVsInlineThenTest.inc

@@ -22,7 +22,11 @@ $closure = function (
     /* testClosureParamTypeNullableInt */
     ?Int $a,
     /* testClosureParamTypeNullableCallable */
-    ?    Callable $b
+    ?    Callable $b,
+    /* testClosureParamTypeNullableStringWithAttributeAndSlashComment */
+    #[AttributeForParam]
+    // This must be a slash or hash comment and the next line must **NOT** have any indentation for the PHP 8.5 deprecation notice (issue PHPCSStandards/PHP_CodeSniffer#1216) to occur.
+?string $c
 /* testClosureReturnTypeNullableInt */
 ) :?INT{};
 

tests/Core/Tokenizers/PHP/NullableVsInlineThenTest.php

@@ -50,16 +50,17 @@ public function testNullable($testMarker)
     public static function dataNullable()
     {
         return [
-            'property declaration, readonly, no visibility'  => ['/* testNullableReadonlyOnly */'],
-            'property declaration, private set'              => ['/* testNullablePrivateSet */'],
-            'property declaration, public and protected set' => ['/* testNullablePublicProtectedSet */'],
-            'property declaration, final, no visibility'     => ['/* testNullableFinalOnly */'],
-            'property declaration, abstract, no visibility'  => ['/* testNullableAbstractOnly */'],
-
-            'closure param type, nullable int'               => ['/* testClosureParamTypeNullableInt */'],
-            'closure param type, nullable callable'          => ['/* testClosureParamTypeNullableCallable */'],
-            'closure return type, nullable int'              => ['/* testClosureReturnTypeNullableInt */'],
-            'function return type, nullable callable'        => ['/* testFunctionReturnTypeNullableCallable */'],
+            'property declaration, readonly, no visibility'                 => ['/* testNullableReadonlyOnly */'],
+            'property declaration, private set'                             => ['/* testNullablePrivateSet */'],
+            'property declaration, public and protected set'                => ['/* testNullablePublicProtectedSet */'],
+            'property declaration, final, no visibility'                    => ['/* testNullableFinalOnly */'],
+            'property declaration, abstract, no visibility'                 => ['/* testNullableAbstractOnly */'],
+
+            'closure param type, nullable int'                              => ['/* testClosureParamTypeNullableInt */'],
+            'closure param type, nullable callable'                         => ['/* testClosureParamTypeNullableCallable */'],
+            'closure param type, nullable string with comment, issue #1216' => ['/* testClosureParamTypeNullableStringWithAttributeAndSlashComment */'],
+            'closure return type, nullable int'                             => ['/* testClosureReturnTypeNullableInt */'],
+            'function return type, nullable callable'                       => ['/* testFunctionReturnTypeNullableCallable */'],
         ];
 
     }//end dataNullable()