Cannot get valid data via kcapi_cipher_dec_aes_cbc for blocks > 64KiB

[apylchagin]

Hello,

I'm experiencing issues with that function for blocks bigger than 64KiB. I'm working on migration from openssl based to kcapi based. The data is coming by 70-80KiB blocks and I need to decrypt them. There is a unit test which just compares the existing openssl based and the new kcapi based implementation. the byte-to-byte comparison is showing that the error exists ONLY in 16 bytes from 0x10000 all other parts of the entire decrypted data are identical.

The environment I've tried on:

1. Stock Ubuntu 20.04 (Intel server platform)
2. NXP imx8 (Android 10)

The quick check I've done by doubling the pagesize in _kcapi_handle_alloc() which avoid split the data into 64KiB blocks and after that the result is exactly the same to openssl output, i.e. all test have passed.

I've build 1.3.1 version by myself for both platforms.

Any guidance or recommendations would be appreciated. can it be that this function is deprecated for some reason?

[smuellerDD]

Am Mittwoch, dem 06.10.2021 um 02:11 -0700 schrieb apylchagin:

Hello, I'm experiencing issues with that function for blocks bigger than 64KiB. I'm working on migration from openssl based to kcapi based. The data is coming by 70-80KiB blocks and I need to decrypt them. There is a unit test which just compares the existing openssl based and the new kcapi based implementation. the byte-to-byte comparison is showing that the error exists ONLY in 16 bytes from 0x10000 all other parts of the entire decrypted data are identical. The environment I've tried on: 1) Stock Ubuntu 20.10 (Intel server platform) 2) NXP imx8 (Android 10) The quick check I've done by doubling the pagesize in _kcapi_handle_alloc() which avoid split the data into 64KiB blocks and after that the result is exactly the same to openssl output. I've build 1.3.1 version by myself for both platforms.

Which algo are you testing? The core issue that I see here is that the algorithm does not support a "steam" operation where multiple input/output calls on the same key/IV are supported. What you did is to increase the allotted buffer to get more data into the kernel in one call where the issue is not triggered. Ciao Stephan

…

[apylchagin]

Thank you for the quick response.
I've got data in AES CBC 128-bit -- I believe this is a function kcapi_cipher_dec_aes_cbc for that algo only -- cbc(aes). right?

The core issue that I see here is that the algorithm does not support a "steam" operation where multiple input/output
calls on the same key/IV are supported.

For my case there is no multiple calls in row of kcapi_cipher_dec_aes_cbc() for the same data-chunk. I've got a single chunk of 80KiB encrypted data and I'm calling kcapi_cipher_dec_aes_cbc() for whole that chunk only once, i.e. ivec is passed only once. The function returns me no errors; it gives exact amount of decrypted bytes, i.e. the same 80KiB.
So, the only multiple calls which I can via trace and kcapi logs are done internally by kcapi.

Per kcapi_cipher_dec_aes_cbc() description I would expect that it should handle unrestricted block sizes with the only requirment to have them 16bytes-aligned. Does it mean that I need to use some other method or mechanism from kcapi, or this is a restriction of CryptoAPI for that algo?

If that function has some restriction it would make sense to add some note about 16*page-size restriction as soon as this is pre-defined algo.

What you did is to increase the allotted buffer to get more data into the kernel in one call where the issue is not triggered.

I understood that. Thanks for confirmation.

[smuellerDD]

Am Mittwoch, dem 06.10.2021 um 03:25 -0700 schrieb apylchagin:

Thank you for the quick response. I've got data in AES CBC 128-bit -- I believe this is a function kcapi_cipher_dec_aes_cbc for that algo only -- cbc(aes). right?

Yes, but there may be many algos behind that. Can you send me the /proc/crypto output?

> The core issue that I see here is that the algorithm does not support a > "steam" operation where multiple input/output calls on the same key/IV are supported. For my case there is no multiple calls in row of kcapi_cipher_dec_aes_cbc() for the same data-chunk.

Maybe so, but libkcapi has to chunk it up commonly after PAGE_SIZE * 16 unless the pipe size is extended. Note, the kernel naturally wants to limit how much data it processes. When, say, you send 1GB in a single submission, the kernel would need to handle/store that 1GB somewhere. As this is not advisable, the processing is chunked up. I try to re-create the issue here.

I've got a single chunk of 80KiB encrypted data and I'm calling kcapi_cipher_dec_aes_cbc() for whole that chunk only once, i.e. ivec is passed only once. The function returns me no errors; it gives exact amount of decrypted bytes, i.e. the same 80KiB. Per kcapi_cipher_dec_aes_cbc() description I would expect that it should handle unrestricted block sizes with the only requirment to have them 16bytes-aligned.  Does it mean that I need to use some other method or mechanism from kcapi, or this is a restriction of CryptoAPI for that also?

There is no size constraint. There maybe constraints in the crypto implementations.

> What you did is to increase the allotted buffer to get more data into the > kernel in one call where the issue is not triggered. I understood that. Thanks for confirmation.

Thanks Stephan

[apylchagin]

Here we go.
crypto.txt.zip -- 18.04
crypto.txt.zip -- 20.04

Just in case I've tried on two ubuntu servers:

1. Ubuntu 18.04.6 LTS -- Linux 4.15.0-159-generic #167-Ubuntu SMP Tue Sep 21 08:55:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2. Ubuntu 20.04.3 LTS -- Linux 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I'm very appreciate your assist.

[smuellerDD]

Am Mittwoch, dem 06.10.2021 um 05:10 -0700 schrieb apylchagin:

Here we go. [crypto.txt.zip]( https://github.com/smuellerDD/libkcapi/files/7293555/crypto.txt.zip)

Thanks. cbc-aes-aesni is the implementation that is used. I will test that here.

[smuellerDD]

Am Mittwoch, 6. Oktober 2021, 12:25:19 CEST schrieb apylchagin: Hi apylchagin,

Thank you for the quick response. I've got data in AES CBC 128-bit -- I believe this is a function kcapi_cipher_dec_aes_cbc for that algo only -- cbc(aes). right? > The core issue that I see here is that the algorithm does not support a > "steam" operation where multiple input/output calls on the same key/IV are supported. For my case there is no multiple calls in row of kcapi_cipher_dec_aes_cbc() for the same data-chunk. I've got a single chunk of 80KiB encrypted data and I'm calling kcapi_cipher_dec_aes_cbc() for whole that chunk only once, i.e. ivec is passed only once. The function returns me no errors; it gives exact amount of decrypted bytes, i.e. the same 80KiB. Per kcapi_cipher_dec_aes_cbc() description I would expect that it should handle unrestricted block sizes with the only requirment to have them 16bytes-aligned. Does it mean that I need to use some other method or mechanism from kcapi, or this is a restriction of CryptoAPI for that also? > What you did is to increase the allotted buffer to get more data into the > kernel in one call where the issue is not triggered. I understood that. Thanks for confirmation.

I am so sorry for the delay. But the LRNG work kept me busy. Now, I performed: - buflen * 5500 == 88000 - dd if=/dev/zero of=file count=1 bs=88000 - kcapi-enc -c "cbc(aes)" -e -i file -o file.out --iv aaaaaaaaaaaaaaaa -- passwd 123 --pbkdfiter 1 - kcapi-enc -c "cbc(aes)" -d -i file.out -o file.2 --iv aaaaaaaaaaaaaaaa -- passwd 123 --pbkdfiter 1 --salt db0897bbb5e4ad830742f1bd33704b45372c2dd6b736fe7623834ee9738785b5 - $ diff file file.2 $ echo $? 0 $ So, the decryption operation works as intended. When I now test the convenience function with the following code, I get a matching output: #include <kcapi.h> #include <stdio.h> #include <string.h> void main(void) { uint8_t key[16] = { 0 }; uint8_t data[88000] = { 0 }; uint8_t data2[88000] = { 1 }; uint8_t ct[88000]; uint8_t iv[16] = { 0 }; if (sizeof(data) % 16) { printf("non-aligned data\n"); return; } kcapi_cipher_enc_aes_cbc(key, sizeof(key), data, sizeof(data), iv, ct, sizeof(ct)); kcapi_cipher_dec_aes_cbc(key, sizeof(key), ct, sizeof(ct), iv, data2, sizeof(data2)); if (memcmp(data, data2, sizeof(data))) { printf("mismatch\n"); } else { printf("match\n"); } } Thus, can you please help me what you do exactly? One more note, the issue you describe is fixed with the change ce5866e When you strace the command above, the key is the following: pipe([4, 5]) = 0 fcntl(4, F_SETPIPE_SZ, 69632) = 131072 fcntl(5, F_SETPIPE_SZ, 69632) = 131072 Thus, if the code above does not work, can you please strace the invocation and check whether the mentioned system calls are present? Thanks Stephan

[apylchagin]

Hello Stephan.

Thank you for your support.

I've spent some time converting my example from bigger project into the dedicated repo https://github.com/apylchagin/kcapi_test. Plz, do not hesitate to use it for verification.

There are 4 possible test cases. KCAPI-ENC_DEC, OPENSSL-ENC_DEC, KCAPI-ENC-OPENSSL-DEC, OPENSSL-ENC-KCAPI-DEC. The code you provided above is a KCAPI-ENC_DEC case and here is the output from strace: EncDec.KcapiEncDec.txt.zip

So, on my setup both cases where openssl and kcapi are in use are failing. Here is the full log: tests.zip

At some moment, by mistake, instead of using own build of 1.3.1 version I used the system-installed in ubuntu 18.04 which is 1.0.3 and all tests surprisingly passed but they finished by error elated to incorrect address for free() call at the very end

Sincerely yours,
Alex P

[smuellerDD]

Am Mittwoch, 27. Oktober 2021, 22:12:19 CEST schrieb apylchagin: Hi apylchagin,

Hello Stephan. I've stent some time converting my example from bigger project into the dedicated repo https://github.com/apylchagin/kcapi_test. Plz, do not hesitate to use it for verification. There are 4 possible test cases. KCAPI-ENC_DEC, OPENSSL-ENC_DEC, KCAPI-ENC-OPENSSL-DEC, OPENSSL-ENC-KCAPI-DEC. The code you provided above is a KCAPI-ENC_DEC case and here is the output from strace: [EncDec.KcapiEncDec.txt.zip](https://github.com/smuellerDD/libkcapi/files/7 429027/EncDec.KcapiEncDec.txt.zip) So, on my setup both cases there openssl and kcapi are in use are failing. By mistake, instead of using own build of 1.3.1 version I used the system-installed in ubuntu 18.04 which is 1.0.3 and all tests passed but they finished by error elated to incorrect address for free() call.

$ mkdir build cmake -G"Unix Makefiles" -B build -S . -- The C compiler identification is GNU 11.2.1 -- The CXX compiler identification is GNU 11.2.1 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.1.1l") CMake Error at CMakeLists.txt:27 (add_subdirectory): The source directory /home/sm/tmp/kcapi_test/gtest does not contain a CMakeLists.txt file.

-- Configuring incomplete, errors occurred! See also "/home/sm/tmp/kcapi_test/build/CMakeFiles/CMakeOutput.log". $ ./setup_and_build.sh ~/tmp/kcapi_test/kcapi/code ~/tmp/kcapi_test autoreconf: 'configure.ac' or 'configure.in' is required ls -l kcapi/code $

--> empty It seems I miss something. Thanks Stephan

Sincerely yours, Alex P

Ciao Stephan

[apylchagin]

Sorry about that. I've added googletest and kcapi repos as a git submodules. thus, you need to clone with submodules update or do it explicitly after clone: git submodule update --init

[smuellerDD]

It sounds like the issue is closed. Thus I close the report. If something is pending, please feel free to reopen it.

[apylchagin]

Sorry but I don’t get it. Could you please advice what version has a fix for it? The label 1.3.1 cannot be used for OpenSSL encoded content. Or the intent that the library can decode the content only encoded by itself? If I’m using some content which came to the software from some other resource I cannot use the cryptoapi?

Thank you for explanation in advance.

[smuellerDD]

Apologies - I misinterpreted your last comment.

After pulling you git repo again, I cannot get the code to configure:

$ cmake -G"Unix Makefiles" -B build -S .
-- The C compiler identification is GNU 11.2.1
-- The CXX compiler identification is GNU 11.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.1.1l")  
CMake Error at CMakeLists.txt:27 (add_subdirectory):
  The source directory

    /home/sm/tmp/kcapi_test/gtest

  does not contain a CMakeLists.txt file.


-- Configuring incomplete, errors occurred!
See also "/home/sm/tmp/kcapi_test/build/CMakeFiles/CMakeOutput.log".

Please provide me with the precise steps how to get your test compiled. Thanks.

[apylchagin]

@smuellerDD sorry for the long delay.
I've update the script in my repo -- now it does git submodules update. Once you run it the build should be done and you should see the tests execution results. the code of test cases is in https://github.com/apylchagin/kcapi_test/blob/main/code/main.cc