Fix eRT decoding: resize buffer to 128 bytes, add UTF-8 validation

[VasylSamoilov]

• Resize eRT buffer from 64 to 128 bytes per IEC 62106-6, fixes eRT buffer not sized for 128 bytes #149
• Add sanitizeUtf8() to handle incomplete multi-byte sequences, fixes No UTF-8 validation before JSON output #150
• Fix off-by-one in RT+/eRT+ tag extraction at string end, fixes RT+/eRT+ tag extraction off-by-one #151

[windytan]

Thanks a lot! Is there any way you could record, in the hexadecimal format, a short snippet of a radio station that transmits such eRT that exhibits the bug? I could then add it to the component or e2e tests. As I don't have eRT stations available locally for testing.

[VasylSamoilov]

I just squashed and pushed new fix.

• Fix eRT+ tag extraction failure with UTF-8 characters, fixes eRT+ tag extraction failure with UTF-8 characters #153

[VasylSamoilov]

Hi. Glad it helped. It's not a radio station as in commercial radio station, I'm writing RDS encoder/decoder based on osmocom-analog radio module, using your decoder as reference decoder. As I'm throwing in different (often invalid) data into redsea, a nice special effects pop up here and there. For this specific case I read iec-62106-6-2018 multiple times to understand what is going on and where we should count bytes, and where we should count characters. Test that caused all the problems was:

		.ert = {
			.enabled = 1,
			.carrier_group = RDS_GROUP_12A,	/* Group 12A for eRT ODA */
			.message = RDS_ERT_3A_MSG_UTF8_LTR_E3,	/* UTF-8 encoding, LTR direction, E3 chartable */
			.text = "The Beatles | Hey Jude | Album: 1967-1970 | Weather: 22°C | Київ",
		},

		.ert_plus = {
			.enabled = 1,
			.carrier_group = RDS_GROUP_13A,	/* Group 13A for eRT+ ODA (different from RT+ on 11A) */
			.message = 0x0000,		/* cb=0, scb=0, template=0 (eRT+ default) */
			.toggle = 0,
			.item_running = 1,
			.tags = {
				{ .content_type = RDS_RTPLUS_CT_INFO_WEATHER, .start = 53, .length = 4 },	/* "22°C" chars 53-56 */
				{ .content_type = RDS_RTPLUS_CT_INFO_OTHER, .start = 60, .length = 4 },		/* "Київ" chars 60-63 */
			},
			.tag_count = 2,
		},

[VasylSamoilov]

If this helps, here is the hex string. I can also capture whatever I'm transmitting with redsea and provide dump, if needed.

DRADIO INFO rds.c:2802 RDS eRT+: Tags cleared
DRADIO INFO rds.c:2793 RDS eRT+: Added tag1 type=25 (info.weather) start=53 len=4
DRADIO INFO rds.c:2793 RDS eRT+: Added tag2 type=30 (info.other) start=60 len=4
DRADIO INFO radio.c:945 RDS eRT+: Enabled on group 26 with 2 tag(s)
DRADIO INFO rds.c:3135 RDS eRT: Set 69 bytes (encoding=UTF-8): "The Beatles | Hey Jude | Album: 1967-1970 | Weather: 22°C | Київ"
DRADIO DEBUG rds.c:3148 RDS eRT DEBUG: length=70, hex dump (first 128 bytes):
54 68 65 20 42 65 61 74 6C 65 73 20 7C 20 48 65
79 20 4A 75 64 65 20 7C 20 41 6C 62 75 6D 3A 20
31 39 36 37 2D 31 39 37 30 20 7C 20 57 65 61 74
68 65 72 3A 20 32 32 C2 B0 43 20 7C 20 D0 9A D0
B8 D1 97 D0 B2 0D
DRADIO INFO radio.c:968 RDS eRT: Enabled on group 24 with 69 bytes (encoding=UTF-8)
DRADIO INFO radio.c:978 RDS Preset: Ukraine (RDS) (PI=6ACE) Groups: 0A/1A/2A

[VasylSamoilov]

PE5PVB's TEF6686_ESP32 https://github.com/PE5PVB/TEF6686_ESP32 based receiver aligns well with the proposed format. The successful decoding of eRT (including eRT+ tags) and proper on-screen display using encoding that match the fixes provided is pretty good validation (the best we can have so far).

[windytan]

Ok thanks, really nice testing

[windytan]

Hi, added just one comment, otherwise I think it's good to merge. I might add tests at a later commit.

[VasylSamoilov]

This commit is mainly meant to illustrate the issue, not to be a final or optimal solution. I addressed your comments and added unit tests as best I could. Please feel free to adjust or improve the implementation if you see any issues.