posix: c_lib_ext: fnmatch: fix escape-oriented regression

A regression in 936d027 introduced a
subtle bug in the way that escaped expressions were handled.

The regression originated with the assumption that test data (originally
adapted from a 3rd-party testsuite) was correct when it was in fact
flawed.

Specifically, `fnmatch("[[?*\\]", "\\", 0)` should fail, since the "\\"
sequence (a single backslash after compilation) escapes the following ']'
character, thus leaving the bracket expression incomplete. Since the
bracket expression (and therefore the whole pattern) is erroneous, the
call should return `FNM_NOMATCH` to indicate failure rather than 0 to
indicate success.

Added new test cases from #98827 and some commentary for subsequent
reviewers.

This change does not complete #55186 but is related to it.

Signed-off-by: Chris Friedt <cfriedt@tenstorrent.com>
Signed-off-by: Jukka Rissanen <jukka.rissanen@nordicsemi.no>
Signed-off-by: Keith Packard <keithp@keithp.com>

Author: cfriedt

lib/posix/c_lib_ext/fnmatch.c

@@ -90,9 +90,7 @@ static const char *rangematch(const char *pattern, int test, int flags)
 		}
 
 		if (c == '\\' && !(flags & FNM_NOESCAPE)) {
-			if (*pattern != ']' && *pattern != EOS) {
-				c = FOLDCASE(*pattern++, flags);
-			}
+			c = FOLDCASE(*pattern++, flags);
 		}
 
 		if (c == EOS) {

tests/posix/c_lib_ext/src/fnmatch.c

@@ -30,8 +30,28 @@ ZTEST(posix_c_lib_ext, test_fnmatch)
 	zassert_equal(fnmatch("a*.c", "a/x.c", FNM_PATHNAME), FNM_NOMATCH);
 	zassert_ok(fnmatch("*/foo", "/foo", FNM_PATHNAME));
 	zassert_ok(fnmatch("-O[01]", "-O1", 0));
-	zassert_ok(fnmatch("[[?*\\]", "\\", 0));
-	zassert_ok(fnmatch("[]?*\\]", "]", 0));
+	/* '\' in pattern escapes ']'. bracket expression is incomplete. error */
+	zassert_equal(fnmatch("[[?*\\]", "\\", 0), FNM_NOMATCH);
+	/* '\' in pattern does not escape ']'. bracket expression complete. '\' matches input '\' */
+	zassert_ok(fnmatch("[[?*\\]", "\\", FNM_NOESCAPE));
+	/* '\' in pattern escapes '\', match '\' */
+	zassert_ok(fnmatch("[[?*\\\\]", "\\", 0));
+	/*
+	 * "[]" (empty bracket expression) is an invalid pattern.
+	 * > The ( ']' ) shall lose its special meaning and represent itself in a bracket expression
+	 * > if it occurs first in the list (after an initial ( '^' ), if any)
+	 * https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_05
+	 *
+	 * So the next test is (again) and incomplete bracket expression and should return error.
+	 * The two tests that follow it also require the ']' to be treated as a literal character to
+	 * match within the bracket expression.
+	 */
+	zassert_equal(fnmatch("[]?*\\]", "]", 0), FNM_NOMATCH);
+	/* '\' in pattern does not escape. bracket expression complete. ']' matches input ']' */
+	zassert_ok(fnmatch("[]?*\\]", "]", FNM_NOESCAPE));
+	/* '\' in pattern escapes '\'. bracket expression complete. ']' matches input ']' */
+	zassert_ok(fnmatch("[]?*\\\\]", "]", 0));
+
 	zassert_ok(fnmatch("[!]a-]", "b", 0));
 	zassert_ok(fnmatch("[]-_]", "^", 0));
 	zassert_ok(fnmatch("[!]-_]", "X", 0));