WolfSSL
/
wolfssl
mirror of https://github.com/wolfSSL/wolfssl.git
1
0
Fork 0
Code Issues Releases Wiki Activity

refactor AESNI implementations and *VECTOR_REGISTERS* macros to allow dynamic as-needed fallback to pure C, via WC_AES_C_DYNAMIC_FALLBACK. 

wolfssl/wolfcrypt/aes.h: add key_C_fallback[] to struct Aes, and remove comment that "AESNI needs key first, rounds 2nd, not sure why yet" now that AES_128_Key_Expansion_AESNI no longer writes rounds after the expanded key.

wolfcrypt/src/aes.c:
* add _AESNI or _aesni suffixes/infixes to AESNI implementations that were missing them: AES_CBC_encrypt(), AES_CBC_decrypt_by*(), AES_ECB_encrypt(), AES_*_Key_Expansion(), AES_set_encrypt_key(), AES_set_decrypt_key(), AES_GCM_encrypt(), AES_GCM_decrypt(), AES_XTS_encrypt(), and AES_XTS_decrypt().
* move key size check from to start of wc_AesSetKeyLocal().
* refactor pure-C AES setkey and cipher implementations to use aes->key_C_fallback when defined(WC_AES_C_DYNAMIC_FALLBACK).
* refactor wc_AesSetKeyLocal() to set up both AESNI and pure-C expanded keys when defined(WC_AES_C_DYNAMIC_FALLBACK).
* refactor all (haveAESNI && aes->use_aesni) conditions to just (aes->use_aesni).
* add macros VECTOR_REGISTERS_PUSH and VECTOR_REGISTERS_POP, which do nothing but push a brace level when !defined(WC_AES_C_DYNAMIC_FALLBACK), but when defined(WC_AES_C_DYNAMIC_FALLBACK), they call SAVE_VECTOR_REGISTERS2() and on failure, temporarily clear aes->use_aesni and restore at _POP().
* refactor all invocations of SAVE_VECTOR_REGISTERS() and RESTORE_VECTOR_REGISTERS() to VECTOR_REGISTERS_PUSH and VECTOR_REGISTERS_POP, except in wc_AesSetKeyLocal(), wc_AesXtsEncrypt(), and wc_AesXtsDecrypt(), which are refactored to use SAVE_VECTOR_REGISTERS2(), with graceful failure concealment if defined(WC_AES_C_DYNAMIC_FALLBACK).
* orthogonalize cleanup code in wc_AesCbcEncrypt(),  wc_AesCcmEncrypt() and wc_AesCcmDecrypt().
* streamline fallthrough software definitions of wc_AesEncryptDirect() and wc_AesDecryptDirect(), and remove special-casing for defined(WOLFSSL_LINUXKM)&&defined(WOLFSSL_AESNI).

wolfcrypt/src/aes_asm.{S,asm}:
* remove errant "movl $10, 240(%rsi)" from AES_128_Key_Expansion_AESNI.
* add _AESNI suffixes/infixes to implementations that needed them.

wolfcrypt/src/{aes_gcm_asm.{S,asm},aes_xts_asm.S}: regenerate from revisions in scripts#357 -- adds _aesni suffixes to implementations that were missing them.

wolfssl/wolfcrypt/types.h: remove DEBUG_VECTOR_REGISTER_ACCESS macros, and add dummy fallthrough definitions for SAVE_VECTOR_REGISTERS2 and WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL.

wolfssl/wolfcrypt/memory.h: adopt DEBUG_VECTOR_REGISTER_ACCESS code from types.h, and add definitions for WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL and WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL.

linuxkm/linuxkm_wc_port.h: add arch-specific macro definitions for SAVE_VECTOR_REGISTERS2().

wolfcrypt/benchmark/benchmark.c: add missing gates around calls to RESTORE_VECTOR_REGISTERS().

configure.ac:
* cover various interdependencies in enable-all/enable-all-crypto, for better behavior in combination with --disable-aesgcm, --disable-ecc, --disable-ocsp, --disable-hmac, --disable-chacha, --disable-ed25519, and --disable-ed448.
* inhibit aesgcm_stream in enable-all/enable-all-crypto when ENABLED_LINUXKM_DEFAULTS, because it is currently incompatible with WC_AES_C_DYNAMIC_FALLBACK.
* add -DWC_AES_C_DYNAMIC_FALLBACK when ENABLED_LINUXKM_DEFAULTS.
* add 3 new interdependency checks: "ECCSI requires ECC.", "SAKKE requires ECC.", "WOLFSSH requires HMAC."

wolfcrypt/src/asn.c: tweak gating to accommodate defined(NO_RSA) && !defined(HAVE_ECC).

wolfcrypt/src/evp.c: tweak gating to accommodate defined(NO_HMAC).

wolfcrypt/src/logging.c: remove DEBUG_VECTOR_REGISTER_ACCESS code (moved to memory.c).

wolfcrypt/src/memory.c: change #include of settings.h to types.h; adopt DEBUG_VECTOR_REGISTER_ACCESS code from logging.c; add implementation of SAVE_VECTOR_REGISTERS2_fuzzer().

wolfcrypt/src/pwdbased.c: add explanatory #error scrypt requires HMAC.

wolfcrypt/test/test.c:
* add DEBUG_VECTOR_REGISTER_ACCESS clauses to aes_xts_128_test(), aesecb_test(), aesctr_test(), aes_test() CBC section, aes256_test() CBC section, and aesgcm_default_test_helper()
* remove duplicate wc_AesEcbDecrypt() in aesecb_test().
* add gating for pbkdf2_test().
* fix cleanup code in dsa_test().
* fix gating in pkcs7authenveloped_run_vectors() to accommodate !defined(HAVE_AESGCM).
* fix gating in cryptocb_test() to accommodate defined(NO_HMAC).

wolfssl/wolfcrypt/cryptocb.h: remove gates around "pk" sub-struct of struct wc_CryptoInfo -- wc_CryptoInfo.pk.type (an int) is used unconditionally when --enable-debug, and is used with DH.

wolfssl/wolfcrypt/error-crypt.h: fix whitespace.
pull/6981/head
Daniel Pouzzner 2023-11-17 01:15:28 -06:00
parent e395aad84b
commit a10260ca5f
20 changed files with 1934 additions and 1345 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

66
configure.ac
View File

@ -720,7 +720,6 @@ then
# this set is also enabled by enable-all-crypto:
test "$enable_atomicuser" = "" && enable_atomicuser=yes
test "$enable_aesgcm" = "" && enable_aesgcm=yes
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_aesccm" = "" && enable_aesccm=yes
test "$enable_aesctr" = "" && enable_aesctr=yes
test "$enable_aeseax" = "" && enable_aeseax=yes
@ -741,22 +740,22 @@ then
test "$enable_hkdf" = "" && enable_hkdf=yes
test "$enable_curve25519" = "" && enable_curve25519=yes
test "$enable_curve448" = "" && enable_curve448=yes
test "$enable_fpecc" = "" && enable_fpecc=yes
test "$enable_eccencrypt" = "" && enable_eccencrypt=yes
test "$enable_fpecc" = "" && test "$enable_ecc" != "no" && enable_fpecc=yes
test "$enable_eccencrypt" = "" && test "$enable_ecc" != "no" && enable_eccencrypt=yes
test "$enable_psk" = "" && enable_psk=yes
test "$enable_cmac" = "" && enable_cmac=yes
test "$enable_siphash" = "" && enable_siphash=yes
test "$enable_xts" = "" && enable_xts=yes
test "$enable_ocsp" = "" && enable_ocsp=yes
test "$enable_ocspstapling" = "" && enable_ocspstapling=yes
test "$enable_ocspstapling2" = "" && enable_ocspstapling2=yes
test "$enable_ocspstapling" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling=yes
test "$enable_ocspstapling2" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling2=yes
test "$enable_crl" = "" && enable_crl=yes
test "$enable_supportedcurves" = "" && enable_supportedcurves=yes
test "$enable_tlsx" = "" && enable_tlsx=yes
test "$enable_pwdbased" = "" && enable_pwdbased=yes
test "$enable_aeskeywrap" = "" && enable_aeskeywrap=yes
test "$enable_x963kdf" = "" && enable_x963kdf=yes
test "$enable_scrypt" = "" && enable_scrypt=yes
test "$enable_scrypt" = "" && test "$enable_hmac" != "no" && enable_scrypt=yes
test "$enable_indef" = "" && enable_indef=yes
test "$enable_enckeys" = "" && enable_enckeys=yes
test "$enable_hashflags" = "" && enable_hashflags=yes
@ -771,7 +770,7 @@ then
test "$enable_md4" = "" && enable_md4=yes
test "$enable_cryptocb" = "" && enable_cryptocb=yes
test "$enable_anon" = "" && enable_anon=yes
test "$enable_ssh" = "" && enable_ssh=yes
test "$enable_ssh" = "" && test "$enable_hmac" != "no" && enable_ssh=yes
test "$enable_savesession" = "" && enable_savesession=yes
test "$enable_savecert" = "" && enable_savecert=yes
@ -797,6 +796,7 @@ then
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
then
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_compkey" = "" && enable_compkey=yes
test "$enable_quic" = "" && enable_quic=yes
AM_CFLAGS="$AM_CFLAGS -DHAVE_CRL_IO -DHAVE_IO_TIMEOUT"
@ -836,7 +836,7 @@ then
if test "$ENABLED_FIPS" = "no"
then
test "$enable_pkcallbacks" = "" && enable_pkcallbacks=yes
test "$enable_xchacha" = "" && enable_xchacha=yes
test "$enable_xchacha" = "" && test "$enable_chacha" != "no" && enable_xchacha=yes
test "$enable_scep" = "" && enable_scep=yes
test "$enable_pkcs7" = "" && enable_pkcs7=yes
test "$enable_nullcipher" = "" && enable_nullcipher=yes
@ -844,9 +844,9 @@ then
if test "$ENABLED_32BIT" != "yes"
then
test "$enable_ed25519" = "" && enable_ed25519=yes
test "$enable_ed25519_stream" = "" && enable_ed25519_stream=yes
test "$enable_ed25519_stream" = "" && test "$enable_ed25519" != "no" && enable_ed25519_stream=yes
test "$enable_ed448" = "" && enable_ed448=yes
test "$enable_ed448_stream" = "" && enable_ed448_stream=yes
test "$enable_ed448_stream" = "" && test "$enable_ed448" != "no" && enable_ed448_stream=yes
fi
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
@ -856,8 +856,8 @@ then
test "$enable_curl" = "" && enable_curl=yes
test "$enable_tcpdump" = "" && enable_tcpdump=yes
test "$enable_eccsi" = "" && enable_eccsi=yes
test "$enable_sakke" = "" && enable_sakke=yes
test "$enable_eccsi" = "" && test "$enable_ecc" != "no" && enable_eccsi=yes
test "$enable_sakke" = "" && test "$enable_ecc" != "no" && enable_sakke=yes
fi
fi
@ -908,7 +908,6 @@ if test "$ENABLED_ALL_CRYPT" = "yes"
then
test "$enable_atomicuser" = "" && enable_atomicuser=yes
test "$enable_aesgcm" = "" && enable_aesgcm=yes
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_aesccm" = "" && enable_aesccm=yes
test "$enable_aesctr" = "" && enable_aesctr=yes
test "$enable_aeseax" = "" && enable_aeseax=yes
@ -929,22 +928,22 @@ then
test "$enable_hkdf" = "" && enable_hkdf=yes
test "$enable_curve25519" = "" && enable_curve25519=yes
test "$enable_curve448" = "" && enable_curve448=yes
test "$enable_fpecc" = "" && enable_fpecc=yes
test "$enable_eccencrypt" = "" && enable_eccencrypt=yes
test "$enable_fpecc" = "" && test "$enable_ecc" != "no" && enable_fpecc=yes
test "$enable_eccencrypt" = "" && test "$enable_ecc" != "no" && enable_eccencrypt=yes
test "$enable_psk" = "" && enable_psk=yes
test "$enable_cmac" = "" && enable_cmac=yes
test "$enable_siphash" = "" && enable_siphash=yes
test "$enable_xts" = "" && enable_xts=yes
test "$enable_ocsp" = "" && enable_ocsp=yes
test "$enable_ocspstapling" = "" && enable_ocspstapling=yes
test "$enable_ocspstapling2" = "" && enable_ocspstapling2=yes
test "$enable_ocspstapling" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling=yes
test "$enable_ocspstapling2" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling2=yes
test "$enable_crl" = "" && enable_crl=yes
test "$enable_supportedcurves" = "" && enable_supportedcurves=yes
test "$enable_tlsx" = "" && enable_tlsx=yes
test "$enable_pwdbased" = "" && enable_pwdbased=yes
test "$enable_aeskeywrap" = "" && enable_aeskeywrap=yes
test "$enable_x963kdf" = "" && enable_x963kdf=yes
test "$enable_scrypt" = "" && enable_scrypt=yes
test "$enable_scrypt" = "" && test "$enable_hmac" != "no" && enable_scrypt=yes
test "$enable_indef" = "" && enable_indef=yes
test "$enable_enckeys" = "" && enable_enckeys=yes
test "$enable_hashflags" = "" && enable_hashflags=yes
@ -959,7 +958,7 @@ then
test "$enable_md4" = "" && enable_md4=yes
test "$enable_cryptocb" = "" && enable_cryptocb=yes
test "$enable_anon" = "" && enable_anon=yes
test "$enable_ssh" = "" && enable_ssh=yes
test "$enable_ssh" = "" && test "$enable_hmac" != "no" && enable_ssh=yes
if test "$ENABLED_32BIT" != "yes"
then
@ -969,6 +968,7 @@ then
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
then
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_compkey" = "" && enable_compkey=yes
fi
@ -983,21 +983,21 @@ then
if test "$ENABLED_FIPS" = "no"
then
test "$enable_pkcallbacks" = "" && enable_pkcallbacks=yes
test "$enable_xchacha" = "" && enable_xchacha=yes
test "$enable_xchacha" = "" && test "$enable_chacha" != "no" && enable_xchacha=yes
test "$enable_pkcs7" = "" && enable_pkcs7=yes
test "$enable_nullcipher" = "" && enable_nullcipher=yes
if test "$ENABLED_32BIT" != "yes"
then
test "$enable_ed25519" = "" && enable_ed25519=yes
test "$enable_ed25519_stream" = "" && enable_ed25519_stream=yes
test "$enable_ed25519_stream" = "" && test "$enable_ed25519" != "no" && enable_ed25519_stream=yes
test "$enable_ed448" = "" && enable_ed448=yes
test "$enable_ed448_stream" = "" && enable_ed448_stream=yes
test "$enable_ed448_stream" = "" && test "$enable_ed448" != "no" && enable_ed448_stream=yes
fi
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
then
test "$enable_eccsi" = "" && enable_eccsi=yes
test "$enable_sakke" = "" && enable_sakke=yes
test "$enable_eccsi" = "" && test "$enable_ecc" != "no" && enable_eccsi=yes
test "$enable_sakke" = "" && test "$enable_ecc" != "no" && enable_sakke=yes
fi
fi
@ -2896,6 +2896,10 @@ then
if test "$ENABLED_AESNI" = "yes" || test "$ENABLED_INTELASM" = "yes"
then
AM_CFLAGS="$AM_CFLAGS -DWOLFSSL_AESNI"
if test "$ENABLED_LINUXKM_DEFAULTS" = "yes"
then
AM_CFLAGS="$AM_CFLAGS -DWC_AES_C_DYNAMIC_FALLBACK"
fi
if test "$CC" != "icc"
then
case $host_os in
@ -3951,6 +3955,10 @@ AC_ARG_ENABLE([eccsi],
if test "x$ENABLED_ECCSI" = "xyes"
then
if test "$ENABLED_ECC" = "no"
then
AC_MSG_ERROR([ECCSI requires ECC.])
fi
AM_CFLAGS="$AM_CFLAGS -DWOLFCRYPT_HAVE_ECCSI -DWOLFSSL_PUBLIC_MP"
fi
@ -3961,6 +3969,11 @@ AC_ARG_ENABLE([sakke],
[ ENABLED_SAKKE=no ]
)
if test "$ENABLED_SAKKE" != "no" && test "$ENABLED_ECC" = "no"
then
AC_MSG_ERROR([SAKKE requires ECC.])
fi
if test "x$ENABLED_SAKKE" = "xsmall"
then
ENABLED_SAKKE="yes"
@ -8926,6 +8939,11 @@ if test "x$ENABLED_OPENSSLCOEXIST" = "xyes"; then
fi
fi
if test "$ENABLED_WOLFSSH" = "yes" && test "$ENABLED_HMAC" = "no"
then
AC_MSG_ERROR([WOLFSSH requires HMAC.])
fi
AS_IF([test "x$ENABLED_WOLFSSH" = "xyes"],[AM_CPPFLAGS="$AM_CPPFLAGS -DWOLFSSL_WOLFSSH"])
# only allow secure renegotiation info with TLSV12 and ASN

2
linuxkm/linuxkm_wc_port.h
View File

@ -185,6 +185,7 @@
#endif
#ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(fail_clause) { int _svr_ret = save_vector_registers_x86(); if (_svr_ret != 0) { fail_clause } }
#define SAVE_VECTOR_REGISTERS2() save_vector_registers_x86()
#endif
#ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() restore_vector_registers_x86()
@ -193,6 +194,7 @@
#include <asm/fpsimd.h>
#ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(fail_clause) { int _svr_ret = save_vector_registers_arm(); if (_svr_ret != 0) { fail_clause } }
#define SAVE_VECTOR_REGISTERS2() save_vector_registers_arm()
#endif
#ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() restore_vector_registers_arm()

4
wolfcrypt/benchmark/benchmark.c
View File

@ -2125,7 +2125,9 @@ static void bench_stats_sym_finish(const char* desc, int useDeviceID,
(void)useDeviceID;
(void)ret;
#ifdef WOLFSSL_LINUXKM_USE_SAVE_VECTOR_REGISTERS
RESTORE_VECTOR_REGISTERS();
#endif
TEST_SLEEP();
} /* bench_stats_sym_finish */
@ -2283,7 +2285,9 @@ static void bench_stats_asym_finish_ex(const char* algo, int strength,
(void)useDeviceID;
(void)ret;
#ifdef WOLFSSL_LINUXKM_USE_SAVE_VECTOR_REGISTERS
RESTORE_VECTOR_REGISTERS();
#endif
TEST_SLEEP();
} /* bench_stats_asym_finish_ex */

1464
wolfcrypt/src/aes.c
View File

File diff suppressed because it is too large Load Diff

161
wolfcrypt/src/aes_asm.S
View File

@ -30,7 +30,7 @@
#ifdef WOLFSSL_X86_64_BUILD
/*
AES_CBC_encrypt (const unsigned char *in,
AES_CBC_encrypt_AESNI (const unsigned char *in,
unsigned char *out,
unsigned char ivec[16],
unsigned long length,
@ -38,11 +38,11 @@ AES_CBC_encrypt (const unsigned char *in,
int nr)
*/
#ifndef __APPLE__
.globl AES_CBC_encrypt
AES_CBC_encrypt:
.globl AES_CBC_encrypt_AESNI
AES_CBC_encrypt_AESNI:
#else
.globl _AES_CBC_encrypt
_AES_CBC_encrypt:
.globl _AES_CBC_encrypt_AESNI
_AES_CBC_encrypt_AESNI:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
@ -95,7 +95,7 @@ ret
#if defined(WOLFSSL_AESNI_BY4)
/*
AES_CBC_decrypt_by4 (const unsigned char *in,
AES_CBC_decrypt_AESNI_by4 (const unsigned char *in,
unsigned char *out,
unsigned char ivec[16],
unsigned long length,
@ -103,11 +103,11 @@ AES_CBC_decrypt_by4 (const unsigned char *in,
int nr)
*/
#ifndef __APPLE__
.globl AES_CBC_decrypt_by4
AES_CBC_decrypt_by4:
.globl AES_CBC_decrypt_AESNI_by4
AES_CBC_decrypt_AESNI_by4:
#else
.globl _AES_CBC_decrypt_by4
_AES_CBC_decrypt_by4:
.globl _AES_CBC_decrypt_AESNI_by4
_AES_CBC_decrypt_AESNI_by4:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
@ -276,7 +276,7 @@ DEND_4:
#elif defined(WOLFSSL_AESNI_BY6)
/*
AES_CBC_decrypt_by6 (const unsigned char *in,
AES_CBC_decrypt_AESNI_by6 (const unsigned char *in,
unsigned char *out,
unsigned char ivec[16],
unsigned long length,
@ -284,11 +284,11 @@ AES_CBC_decrypt_by6 (const unsigned char *in,
int nr)
*/
#ifndef __APPLE__
.globl AES_CBC_decrypt_by6
AES_CBC_decrypt_by6:
.globl AES_CBC_decrypt_AESNI_by6
AES_CBC_decrypt_AESNI_by6:
#else
.globl _AES_CBC_decrypt_by6
_AES_CBC_decrypt_by6:
.globl _AES_CBC_decrypt_AESNI_by6
_AES_CBC_decrypt_AESNI_by6:
#endif
# parameter 1: %rdi - in
# parameter 2: %rsi - out
@ -504,7 +504,7 @@ DEND_6:
#else /* WOLFSSL_AESNI_BYx */
/*
AES_CBC_decrypt_by8 (const unsigned char *in,
AES_CBC_decrypt_AESNI_by8 (const unsigned char *in,
unsigned char *out,
unsigned char ivec[16],
unsigned long length,
@ -512,11 +512,11 @@ AES_CBC_decrypt_by8 (const unsigned char *in,
int nr)
*/
#ifndef __APPLE__
.globl AES_CBC_decrypt_by8
AES_CBC_decrypt_by8:
.globl AES_CBC_decrypt_AESNI_by8
AES_CBC_decrypt_AESNI_by8:
#else
.globl _AES_CBC_decrypt_by8
_AES_CBC_decrypt_by8:
.globl _AES_CBC_decrypt_AESNI_by8
_AES_CBC_decrypt_AESNI_by8:
#endif
# parameter 1: %rdi - in
# parameter 2: %rsi - out
@ -761,18 +761,18 @@ DEND_8:
/*
AES_ECB_encrypt (const unsigned char *in,
AES_ECB_encrypt_AESNI (const unsigned char *in,
unsigned char *out,
unsigned long length,
const unsigned char *KS,
int nr)
*/
#ifndef __APPLE__
.globl AES_ECB_encrypt
AES_ECB_encrypt:
.globl AES_ECB_encrypt_AESNI
AES_ECB_encrypt_AESNI:
#else
.globl _AES_ECB_encrypt
_AES_ECB_encrypt:
.globl _AES_ECB_encrypt_AESNI
_AES_ECB_encrypt_AESNI:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
@ -925,18 +925,18 @@ EECB_END_4:
/*
AES_ECB_decrypt (const unsigned char *in,
AES_ECB_decrypt_AESNI (const unsigned char *in,
unsigned char *out,
unsigned long length,
const unsigned char *KS,
int nr)
*/
#ifndef __APPLE__
.globl AES_ECB_decrypt
AES_ECB_decrypt:
.globl AES_ECB_decrypt_AESNI
AES_ECB_decrypt_AESNI:
#else
.globl _AES_ECB_decrypt
_AES_ECB_decrypt:
.globl _AES_ECB_decrypt_AESNI
_AES_ECB_decrypt_AESNI:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
@ -1092,20 +1092,19 @@ DECB_END_4:
/*
void AES_128_Key_Expansion(const unsigned char* userkey,
void AES_128_Key_Expansion_AESNI(const unsigned char* userkey,
unsigned char* key_schedule);
*/
.align 16,0x90
#ifndef __APPLE__
.globl AES_128_Key_Expansion
AES_128_Key_Expansion:
.globl AES_128_Key_Expansion_AESNI
AES_128_Key_Expansion_AESNI:
#else
.globl _AES_128_Key_Expansion
_AES_128_Key_Expansion:
.globl _AES_128_Key_Expansion_AESNI
_AES_128_Key_Expansion_AESNI:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
movl $10, 240(%rsi)
movdqu (%rdi), %xmm1
movdqa %xmm1, (%rsi)
@ -1158,15 +1157,15 @@ ret
/*
void AES_192_Key_Expansion (const unsigned char *userkey,
void AES_192_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key)
*/
#ifndef __APPLE__
.globl AES_192_Key_Expansion
AES_192_Key_Expansion:
.globl AES_192_Key_Expansion_AESNI
AES_192_Key_Expansion_AESNI:
#else
.globl _AES_192_Key_Expansion
_AES_192_Key_Expansion:
.globl _AES_192_Key_Expansion_AESNI
_AES_192_Key_Expansion_AESNI:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
@ -1249,15 +1248,15 @@ ret
/*
void AES_256_Key_Expansion (const unsigned char *userkey,
void AES_256_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key)
*/
#ifndef __APPLE__
.globl AES_256_Key_Expansion
AES_256_Key_Expansion:
.globl AES_256_Key_Expansion_AESNI
AES_256_Key_Expansion_AESNI:
#else
.globl _AES_256_Key_Expansion
_AES_256_Key_Expansion:
.globl _AES_256_Key_Expansion_AESNI
_AES_256_Key_Expansion_AESNI:
#endif
# parameter 1: %rdi
# parameter 2: %rsi
@ -1337,7 +1336,7 @@ ret
#elif defined WOLFSSL_X86_BUILD
/*
AES_CBC_encrypt (const unsigned char *in,
AES_CBC_encrypt_AESNI (const unsigned char *in,
unsigned char *out,
unsigned char ivec[16],
unsigned long length,
@ -1345,11 +1344,11 @@ AES_CBC_encrypt (const unsigned char *in,
int nr)
*/
#ifndef __APPLE__
.globl AES_CBC_encrypt
AES_CBC_encrypt:
.globl AES_CBC_encrypt_AESNI
AES_CBC_encrypt_AESNI:
#else
.globl _AES_CBC_encrypt
_AES_CBC_encrypt:
.globl _AES_CBC_encrypt_AESNI
_AES_CBC_encrypt_AESNI:
#endif
# parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi
@ -1416,7 +1415,7 @@ _AES_CBC_encrypt:
/*
AES_CBC_decrypt_by4 (const unsigned char *in,
AES_CBC_decrypt_AESNI_by4 (const unsigned char *in,
unsigned char *out,
unsigned char ivec[16],
unsigned long length,
@ -1424,11 +1423,11 @@ AES_CBC_decrypt_by4 (const unsigned char *in,
int nr)
*/
#ifndef __APPLE__
.globl AES_CBC_decrypt_by4
AES_CBC_decrypt_by4:
.globl AES_CBC_decrypt_AESNI_by4
AES_CBC_decrypt_AESNI_by4:
#else
.globl _AES_CBC_decrypt_by4
_AES_CBC_decrypt_by4:
.globl _AES_CBC_decrypt_AESNI_by4
_AES_CBC_decrypt_AESNI_by4:
#endif
# parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi
@ -1614,18 +1613,18 @@ DEND_4:
ret
/*
AES_ECB_encrypt (const unsigned char *in,
AES_ECB_encrypt_AESNI (const unsigned char *in,
unsigned char *out,
unsigned long length,
const unsigned char *KS,
int nr)
*/
#ifndef __APPLE__
.globl AES_ECB_encrypt
AES_ECB_encrypt:
.globl AES_ECB_encrypt_AESNI
AES_ECB_encrypt_AESNI:
#else
.globl _AES_ECB_encrypt
_AES_ECB_encrypt:
.globl _AES_ECB_encrypt_AESNI
_AES_ECB_encrypt_AESNI:
#endif
# parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi
@ -1791,18 +1790,18 @@ EECB_END_4:
/*
AES_ECB_decrypt (const unsigned char *in,
AES_ECB_decrypt_AESNI (const unsigned char *in,
unsigned char *out,
unsigned long length,
const unsigned char *KS,
int nr)
*/
#ifndef __APPLE__
.globl AES_ECB_decrypt
AES_ECB_decrypt:
.globl AES_ECB_decrypt_AESNI
AES_ECB_decrypt_AESNI:
#else
.globl _AES_ECB_decrypt
_AES_ECB_decrypt:
.globl _AES_ECB_decrypt_AESNI
_AES_ECB_decrypt_AESNI:
#endif
# parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi
@ -1969,16 +1968,16 @@ DECB_END_4:
/*
void AES_128_Key_Expansion(const unsigned char* userkey,
void AES_128_Key_Expansion_AESNI(const unsigned char* userkey,
unsigned char* key_schedule);
*/
.align 16,0x90
#ifndef __APPLE__
.globl AES_128_Key_Expansion
AES_128_Key_Expansion:
.globl AES_128_Key_Expansion_AESNI
AES_128_Key_Expansion_AESNI:
#else
.globl _AES_128_Key_Expansion
_AES_128_Key_Expansion:
.globl _AES_128_Key_Expansion_AESNI
_AES_128_Key_Expansion_AESNI:
#endif
# parameter 1: stack[4] => %eax
# parameter 2: stack[8] => %edx
@ -2038,15 +2037,15 @@ PREPARE_ROUNDKEY_128:
/*
void AES_192_Key_Expansion (const unsigned char *userkey,
void AES_192_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key)
*/
#ifndef __APPLE__
.globl AES_192_Key_Expansion
AES_192_Key_Expansion:
.globl AES_192_Key_Expansion_AESNI
AES_192_Key_Expansion_AESNI:
#else
.globl _AES_192_Key_Expansion
_AES_192_Key_Expansion:
.globl _AES_192_Key_Expansion_AESNI
_AES_192_Key_Expansion_AESNI:
#endif
# parameter 1: stack[4] => %eax
# parameter 2: stack[8] => %edx
@ -2131,15 +2130,15 @@ PREPARE_ROUNDKEY_192:
/*
void AES_256_Key_Expansion (const unsigned char *userkey,
void AES_256_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key)
*/
#ifndef __APPLE__
.globl AES_256_Key_Expansion
AES_256_Key_Expansion:
.globl AES_256_Key_Expansion_AESNI
AES_256_Key_Expansion_AESNI:
#else
.globl _AES_256_Key_Expansion
_AES_256_Key_Expansion:
.globl _AES_256_Key_Expansion_AESNI
_AES_256_Key_Expansion_AESNI:
#endif
# parameter 1: stack[4] => %eax
# parameter 2: stack[8] => %edx

50
wolfcrypt/src/aes_asm.asm
View File

@ -47,14 +47,14 @@ ENDIF
; /*
; AES_CBC_encrypt[const ,unsigned char*in
; AES_CBC_encrypt_AESNI[const ,unsigned char*in
; unsigned ,char*out
; unsigned ,char ivec+16
; unsigned ,long length
; const ,unsigned char*KS
; int nr]
; */
AES_CBC_encrypt PROC
AES_CBC_encrypt_AESNI PROC
;# parameter 1: rdi
;# parameter 2: rsi
;# parameter 3: rdx
@ -117,16 +117,16 @@ LAST:
mov rdi,rax
mov rsi,r11
ret
AES_CBC_encrypt ENDP
AES_CBC_encrypt_AESNI ENDP
; void AES_CBC_decrypt_by4(const unsigned char* in,
; void AES_CBC_decrypt_AESNI_by4(const unsigned char* in,
; unsigned char* out,
; unsigned char ivec[16],
; unsigned long length,
; const unsigned char* KS,
; int nr)
AES_CBC_decrypt_by4 PROC
AES_CBC_decrypt_AESNI_by4 PROC
; parameter 1: rdi
; parameter 2: rsi
; parameter 3: rdx
@ -325,16 +325,16 @@ DEND_4:
movdqa xmm15, [rsp+112]
add rsp, 8+8*16 ; 8 = align stack , 8 xmm6-12,15 16 bytes each
ret
AES_CBC_decrypt_by4 ENDP
AES_CBC_decrypt_AESNI_by4 ENDP
; void AES_CBC_decrypt_by6(const unsigned char *in,
; void AES_CBC_decrypt_AESNI_by6(const unsigned char *in,
; unsigned char *out,
; unsigned char ivec[16],
; unsigned long length,
; const unsigned char *KS,
; int nr)
AES_CBC_decrypt_by6 PROC
AES_CBC_decrypt_AESNI_by6 PROC
; parameter 1: rdi - in
; parameter 2: rsi - out
; parameter 3: rdx - ivec
@ -582,16 +582,16 @@ DEND_6:
movdqa xmm14, [rsp+128]
add rsp, 8+9*16 ; 8 = align stack , 9 xmm6-14 16 bytes each
ret
AES_CBC_decrypt_by6 ENDP
AES_CBC_decrypt_AESNI_by6 ENDP
; void AES_CBC_decrypt_by8(const unsigned char *in,
; void AES_CBC_decrypt_AESNI_by8(const unsigned char *in,
; unsigned char *out,
; unsigned char ivec[16],
; unsigned long length,
; const unsigned char *KS,
; int nr)
AES_CBC_decrypt_by8 PROC
AES_CBC_decrypt_AESNI_by8 PROC
; parameter 1: rdi - in
; parameter 2: rsi - out
; parameter 3: rdx - ivec
@ -865,18 +865,18 @@ DEND_8:
movdqa xmm13, [rsp+112]
add rsp, 8+8*16 ; 8 = align stack , 8 xmm6-13 16 bytes each
ret
AES_CBC_decrypt_by8 ENDP
AES_CBC_decrypt_AESNI_by8 ENDP
; /*
; AES_ECB_encrypt[const ,unsigned char*in
; AES_ECB_encrypt_AESNI[const ,unsigned char*in
; unsigned ,char*out
; unsigned ,long length
; const ,unsigned char*KS
; int nr]
; */
; . globl AES_ECB_encrypt
AES_ECB_encrypt PROC
AES_ECB_encrypt_AESNI PROC
;# parameter 1: rdi
;# parameter 2: rsi
;# parameter 3: rdx
@ -1054,7 +1054,7 @@ EECB_END_4:
movdqa xmm12, [rsp+48]
add rsp,8+4*16 ; 8 = align stack , 4 xmm9-12 16 bytes each
ret
AES_ECB_encrypt ENDP
AES_ECB_encrypt_AESNI ENDP
; /*
; AES_ECB_decrypt[const ,unsigned char*in
@ -1241,17 +1241,17 @@ DECB_END_4:
movdqa xmm12, [rsp+48]
add rsp,8+4*16 ; 8 = align stack , 4 xmm9-12 16 bytes each
ret
AES_ECB_decrypt ENDP
AES_ECB_decrypt_AESNI ENDP
; /*
; void ,AES_128_Key_Expansion[const unsigned char*userkey
; void ,AES_128_Key_Expansion_AESNI[const unsigned char*userkey
; unsigned char*key_schedule]/
; */
; . align 16,0x90
; . globl AES_128_Key_Expansion
AES_128_Key_Expansion PROC
AES_128_Key_Expansion_AESNI PROC
;# parameter 1: rdi
;# parameter 2: rsi
@ -1322,14 +1322,14 @@ PREPARE_ROUNDKEY_128:
pxor xmm1,xmm3
pxor xmm1,xmm2
ret
AES_128_Key_Expansion ENDP
AES_128_Key_Expansion_AESNI ENDP
; /*
; void ,AES_192_Key_Expansion[const unsigned char*userkey
; void ,AES_192_Key_Expansion_AESNI[const unsigned char*userkey
; unsigned char*key]
; */
; . globl AES_192_Key_Expansion
AES_192_Key_Expansion PROC
AES_192_Key_Expansion_AESNI PROC
;# parameter 1: rdi
;# parameter 2: rsi
@ -1426,14 +1426,14 @@ PREPARE_ROUNDKEY_192:
pxor xmm3,xmm4
pxor xmm3,xmm2
ret
AES_192_Key_Expansion ENDP
AES_192_Key_Expansion_AESNI ENDP
; /*
; void ,AES_256_Key_Expansion[const unsigned char*userkey
; void ,AES_256_Key_Expansion_AESNI[const unsigned char*userkey
; unsigned char*key]
; */
; . globl AES_256_Key_Expansion
AES_256_Key_Expansion PROC
AES_256_Key_Expansion_AESNI PROC
;# parameter 1: rdi
;# parameter 2: rsi
@ -1495,7 +1495,7 @@ AES_256_Key_Expansion PROC
mov rdi,rax
mov rsi,r11
ret
AES_256_Key_Expansion ENDP
AES_256_Key_Expansion_AESNI ENDP
MAKE_RK256_a:
pshufd xmm2,xmm2,0ffh

292
wolfcrypt/src/aes_gcm_asm.S
View File

@ -180,15 +180,15 @@ L_aes_gcm_mod2_128:
.quad 0x1, 0xc200000000000000
#ifndef __APPLE__
.text
.globl AES_GCM_encrypt
.type AES_GCM_encrypt,@function
.globl AES_GCM_encrypt_aesni
.type AES_GCM_encrypt_aesni,@function
.align 16
AES_GCM_encrypt:
AES_GCM_encrypt_aesni:
#else
.section __TEXT,__text
.globl _AES_GCM_encrypt
.globl _AES_GCM_encrypt_aesni
.p2align 4
_AES_GCM_encrypt:
_AES_GCM_encrypt_aesni:
#endif /* __APPLE__ */
pushq %r13
pushq %r12
@ -207,7 +207,7 @@ _AES_GCM_encrypt:
pxor %xmm6, %xmm6
cmpl $12, %ebx
movl %ebx, %edx
jne L_AES_GCM_encrypt_iv_not_12
jne L_AES_GCM_encrypt_aesni_iv_not_12
# # Calculate values when IV is 12 bytes
# Set counter based on IV
movl $0x1000000, %ecx
@ -247,7 +247,7 @@ _AES_GCM_encrypt:
aesenc %xmm7, %xmm1
cmpl $11, %r10d
movdqa 160(%r15), %xmm7
jl L_AES_GCM_encrypt_calc_iv_12_last
jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1
movdqa 176(%r15), %xmm7
@ -255,20 +255,20 @@ _AES_GCM_encrypt:
aesenc %xmm7, %xmm1
cmpl $13, %r10d
movdqa 192(%r15), %xmm7
jl L_AES_GCM_encrypt_calc_iv_12_last
jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1
movdqa 208(%r15), %xmm7
aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1
movdqa 224(%r15), %xmm7
L_AES_GCM_encrypt_calc_iv_12_last:
L_AES_GCM_encrypt_aesni_calc_iv_12_last:
aesenclast %xmm7, %xmm5
aesenclast %xmm7, %xmm1
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
movdqu %xmm1, 144(%rsp)
jmp L_AES_GCM_encrypt_iv_done
L_AES_GCM_encrypt_iv_not_12:
jmp L_AES_GCM_encrypt_aesni_iv_done
L_AES_GCM_encrypt_aesni_iv_not_12:
# Calculate values when IV is not 12 bytes
# H = Encrypt X(=0)
movdqa (%r15), %xmm5
@ -283,27 +283,27 @@ L_AES_GCM_encrypt_iv_not_12:
aesenc 144(%r15), %xmm5
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5
aesenc 176(%r15), %xmm5
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5
aesenc 208(%r15), %xmm5
movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast %xmm9, %xmm5
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
# Calc counter
# Initialization vector
cmpl $0x00, %edx
movq $0x00, %rcx
je L_AES_GCM_encrypt_calc_iv_done
je L_AES_GCM_encrypt_aesni_calc_iv_done
cmpl $16, %edx
jl L_AES_GCM_encrypt_calc_iv_lt16
jl L_AES_GCM_encrypt_aesni_calc_iv_lt16
andl $0xfffffff0, %edx
L_AES_GCM_encrypt_calc_iv_16_loop:
L_AES_GCM_encrypt_aesni_calc_iv_16_loop:
movdqu (%rax,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm4
@ -363,22 +363,22 @@ L_AES_GCM_encrypt_calc_iv_16_loop:
pxor %xmm2, %xmm4
addl $16, %ecx
cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_iv_16_loop
jl L_AES_GCM_encrypt_aesni_calc_iv_16_loop
movl %ebx, %edx
cmpl %edx, %ecx
je L_AES_GCM_encrypt_calc_iv_done
L_AES_GCM_encrypt_calc_iv_lt16:
je L_AES_GCM_encrypt_aesni_calc_iv_done
L_AES_GCM_encrypt_aesni_calc_iv_lt16:
subq $16, %rsp
pxor %xmm8, %xmm8
xorl %ebx, %ebx
movdqu %xmm8, (%rsp)
L_AES_GCM_encrypt_calc_iv_loop:
L_AES_GCM_encrypt_aesni_calc_iv_loop:
movzbl (%rax,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1)
incl %ecx
incl %ebx
cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_iv_loop
jl L_AES_GCM_encrypt_aesni_calc_iv_loop
movdqu (%rsp), %xmm8
addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -437,7 +437,7 @@ L_AES_GCM_encrypt_calc_iv_loop:
pxor %xmm1, %xmm2
pxor %xmm7, %xmm2
pxor %xmm2, %xmm4
L_AES_GCM_encrypt_calc_iv_done:
L_AES_GCM_encrypt_aesni_calc_iv_done:
# T = Encrypt counter
pxor %xmm0, %xmm0
shll $3, %edx
@ -512,28 +512,28 @@ L_AES_GCM_encrypt_calc_iv_done:
aesenc 144(%r15), %xmm8
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast %xmm9, %xmm8
movdqu %xmm8, 144(%rsp)
L_AES_GCM_encrypt_iv_done:
L_AES_GCM_encrypt_aesni_iv_done:
# Additional authentication data
movl %r11d, %edx
cmpl $0x00, %edx
je L_AES_GCM_encrypt_calc_aad_done
je L_AES_GCM_encrypt_aesni_calc_aad_done
xorl %ecx, %ecx
cmpl $16, %edx
jl L_AES_GCM_encrypt_calc_aad_lt16
jl L_AES_GCM_encrypt_aesni_calc_aad_lt16
andl $0xfffffff0, %edx
L_AES_GCM_encrypt_calc_aad_16_loop:
L_AES_GCM_encrypt_aesni_calc_aad_16_loop:
movdqu (%r12,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm6
@ -593,22 +593,22 @@ L_AES_GCM_encrypt_calc_aad_16_loop:
pxor %xmm2, %xmm6
addl $16, %ecx
cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_aad_16_loop
jl L_AES_GCM_encrypt_aesni_calc_aad_16_loop
movl %r11d, %edx
cmpl %edx, %ecx
je L_AES_GCM_encrypt_calc_aad_done
L_AES_GCM_encrypt_calc_aad_lt16:
je L_AES_GCM_encrypt_aesni_calc_aad_done
L_AES_GCM_encrypt_aesni_calc_aad_lt16:
subq $16, %rsp
pxor %xmm8, %xmm8
xorl %ebx, %ebx
movdqu %xmm8, (%rsp)
L_AES_GCM_encrypt_calc_aad_loop:
L_AES_GCM_encrypt_aesni_calc_aad_loop:
movzbl (%r12,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1)
incl %ecx
incl %ebx
cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_aad_loop
jl L_AES_GCM_encrypt_aesni_calc_aad_loop
movdqu (%rsp), %xmm8
addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -667,7 +667,7 @@ L_AES_GCM_encrypt_calc_aad_loop:
pxor %xmm1, %xmm2
pxor %xmm7, %xmm2
pxor %xmm2, %xmm6
L_AES_GCM_encrypt_calc_aad_done:
L_AES_GCM_encrypt_aesni_calc_aad_done:
# Calculate counter and H
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
movdqa %xmm5, %xmm9
@ -685,7 +685,7 @@ L_AES_GCM_encrypt_calc_aad_done:
xorq %rbx, %rbx
cmpl $0x80, %r9d
movl %r9d, %r13d
jl L_AES_GCM_encrypt_done_128
jl L_AES_GCM_encrypt_aesni_done_128
andl $0xffffff80, %r13d
movdqa %xmm6, %xmm2
# H ^ 1
@ -1104,7 +1104,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc %xmm7, %xmm15
cmpl $11, %r10d
movdqa 160(%r15), %xmm7
jl L_AES_GCM_encrypt_enc_done
jl L_AES_GCM_encrypt_aesni_enc_done
aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10
@ -1124,7 +1124,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc %xmm7, %xmm15
cmpl $13, %r10d
movdqa 192(%r15), %xmm7
jl L_AES_GCM_encrypt_enc_done
jl L_AES_GCM_encrypt_aesni_enc_done
aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10
@ -1143,7 +1143,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc %xmm7, %xmm14
aesenc %xmm7, %xmm15
movdqa 224(%r15), %xmm7
L_AES_GCM_encrypt_enc_done:
L_AES_GCM_encrypt_aesni_enc_done:
aesenclast %xmm7, %xmm8
aesenclast %xmm7, %xmm9
movdqu (%rdi), %xmm0
@ -1178,9 +1178,9 @@ L_AES_GCM_encrypt_enc_done:
movdqu %xmm15, 112(%rsi)
cmpl $0x80, %r13d
movl $0x80, %ebx
jle L_AES_GCM_encrypt_end_128
jle L_AES_GCM_encrypt_aesni_end_128
# More 128 bytes of input
L_AES_GCM_encrypt_ghash_128:
L_AES_GCM_encrypt_aesni_ghash_128:
leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8
@ -1448,7 +1448,7 @@ L_AES_GCM_encrypt_ghash_128:
pxor %xmm3, %xmm2
cmpl $11, %r10d
movdqa 160(%r15), %xmm7
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10
@ -1468,7 +1468,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc %xmm7, %xmm15
cmpl $13, %r10d
movdqa 192(%r15), %xmm7
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10
@ -1487,7 +1487,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc %xmm7, %xmm14
aesenc %xmm7, %xmm15
movdqa 224(%r15), %xmm7
L_AES_GCM_encrypt_aesenc_128_ghash_avx_done:
L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast %xmm7, %xmm8
aesenclast %xmm7, %xmm9
movdqu (%rcx), %xmm0
@ -1522,8 +1522,8 @@ L_AES_GCM_encrypt_aesenc_128_ghash_avx_done:
movdqu %xmm15, 112(%rdx)
addl $0x80, %ebx
cmpl %r13d, %ebx
jl L_AES_GCM_encrypt_ghash_128
L_AES_GCM_encrypt_end_128:
jl L_AES_GCM_encrypt_aesni_ghash_128
L_AES_GCM_encrypt_aesni_end_128:
movdqa L_aes_gcm_bswap_mask(%rip), %xmm4
pshufb %xmm4, %xmm8
pshufb %xmm4, %xmm9
@ -1710,14 +1710,14 @@ L_AES_GCM_encrypt_end_128:
pxor %xmm4, %xmm2
pxor %xmm2, %xmm6
movdqu (%rsp), %xmm5
L_AES_GCM_encrypt_done_128:
L_AES_GCM_encrypt_aesni_done_128:
movl %r9d, %edx
cmpl %edx, %ebx
jge L_AES_GCM_encrypt_done_enc
jge L_AES_GCM_encrypt_aesni_done_enc
movl %r9d, %r13d
andl $0xfffffff0, %r13d
cmpl %r13d, %ebx
jge L_AES_GCM_encrypt_last_block_done
jge L_AES_GCM_encrypt_aesni_last_block_done
leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8
@ -1737,16 +1737,16 @@ L_AES_GCM_encrypt_done_128:
aesenc 144(%r15), %xmm8
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last:
aesenclast %xmm9, %xmm8
movdqu (%rcx), %xmm9
pxor %xmm9, %xmm8
@ -1755,8 +1755,8 @@ L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last:
pxor %xmm8, %xmm6
addl $16, %ebx
cmpl %r13d, %ebx
jge L_AES_GCM_encrypt_last_block_ghash
L_AES_GCM_encrypt_last_block_start:
jge L_AES_GCM_encrypt_aesni_last_block_ghash
L_AES_GCM_encrypt_aesni_last_block_start:
leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8
@ -1801,16 +1801,16 @@ L_AES_GCM_encrypt_last_block_start:
pxor %xmm3, %xmm6
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_gfmul_last
jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_gfmul_last
jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_aesenc_gfmul_last:
L_AES_GCM_encrypt_aesni_aesenc_gfmul_last:
aesenclast %xmm9, %xmm8
movdqu (%rcx), %xmm9
pxor %xmm9, %xmm8
@ -1819,8 +1819,8 @@ L_AES_GCM_encrypt_aesenc_gfmul_last:
pxor %xmm8, %xmm6
addl $16, %ebx
cmpl %r13d, %ebx
jl L_AES_GCM_encrypt_last_block_start
L_AES_GCM_encrypt_last_block_ghash:
jl L_AES_GCM_encrypt_aesni_last_block_start
L_AES_GCM_encrypt_aesni_last_block_ghash:
pshufd $0x4e, %xmm5, %xmm9
pshufd $0x4e, %xmm6, %xmm10
movdqa %xmm6, %xmm11
@ -1861,11 +1861,11 @@ L_AES_GCM_encrypt_last_block_ghash:
pxor %xmm13, %xmm14
pxor %xmm8, %xmm14
pxor %xmm14, %xmm6
L_AES_GCM_encrypt_last_block_done:
L_AES_GCM_encrypt_aesni_last_block_done:
movl %r9d, %ecx
movl %ecx, %edx
andl $15, %ecx
jz L_AES_GCM_encrypt_aesenc_last15_enc_avx_done
jz L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done
movdqu 128(%rsp), %xmm4
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
pxor (%r15), %xmm4
@ -1880,21 +1880,21 @@ L_AES_GCM_encrypt_last_block_done:
aesenc 144(%r15), %xmm4
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc %xmm9, %xmm4
aesenc 176(%r15), %xmm4
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc %xmm9, %xmm4
aesenc 208(%r15), %xmm4
movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last:
aesenclast %xmm9, %xmm4
subq $16, %rsp
xorl %ecx, %ecx
movdqu %xmm4, (%rsp)
L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop:
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop:
movzbl (%rdi,%rbx,1), %r13d
xorb (%rsp,%rcx,1), %r13b
movb %r13b, (%rsi,%rbx,1)
@ -1902,16 +1902,16 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop:
incl %ebx
incl %ecx
cmpl %edx, %ebx
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop
xorq %r13, %r13
cmpl $16, %ecx
je L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc
L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop:
je L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop:
movb %r13b, (%rsp,%rcx,1)
incl %ecx
cmpl $16, %ecx
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop
L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc:
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc:
movdqu (%rsp), %xmm4
addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm4
@ -1956,8 +1956,8 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc:
pxor %xmm13, %xmm14
pxor %xmm8, %xmm14
pxor %xmm14, %xmm6
L_AES_GCM_encrypt_aesenc_last15_enc_avx_done:
L_AES_GCM_encrypt_done_enc:
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done:
L_AES_GCM_encrypt_aesni_done_enc:
movl %r9d, %edx
movl %r11d, %ecx
shlq $3, %rdx
@ -2009,19 +2009,19 @@ L_AES_GCM_encrypt_done_enc:
movdqu 144(%rsp), %xmm0
pxor %xmm6, %xmm0
cmpl $16, %r14d
je L_AES_GCM_encrypt_store_tag_16
je L_AES_GCM_encrypt_aesni_store_tag_16
xorq %rcx, %rcx
movdqu %xmm0, (%rsp)
L_AES_GCM_encrypt_store_tag_loop:
L_AES_GCM_encrypt_aesni_store_tag_loop:
movzbl (%rsp,%rcx,1), %r13d
movb %r13b, (%r8,%rcx,1)
incl %ecx
cmpl %r14d, %ecx
jne L_AES_GCM_encrypt_store_tag_loop
jmp L_AES_GCM_encrypt_store_tag_done
L_AES_GCM_encrypt_store_tag_16:
jne L_AES_GCM_encrypt_aesni_store_tag_loop
jmp L_AES_GCM_encrypt_aesni_store_tag_done
L_AES_GCM_encrypt_aesni_store_tag_16:
movdqu %xmm0, (%r8)
L_AES_GCM_encrypt_store_tag_done:
L_AES_GCM_encrypt_aesni_store_tag_done:
addq $0xa0, %rsp
popq %r15
popq %r14
@ -2030,19 +2030,19 @@ L_AES_GCM_encrypt_store_tag_done:
popq %r13
repz retq
#ifndef __APPLE__
.size AES_GCM_encrypt,.-AES_GCM_encrypt
.size AES_GCM_encrypt_aesni,.-AES_GCM_encrypt_aesni
#endif /* __APPLE__ */
#ifndef __APPLE__
.text
.globl AES_GCM_decrypt
.type AES_GCM_decrypt,@function
.globl AES_GCM_decrypt_aesni
.type AES_GCM_decrypt_aesni,@function
.align 16
AES_GCM_decrypt:
AES_GCM_decrypt_aesni:
#else
.section __TEXT,__text
.globl _AES_GCM_decrypt
.globl _AES_GCM_decrypt_aesni
.p2align 4
_AES_GCM_decrypt:
_AES_GCM_decrypt_aesni:
#endif /* __APPLE__ */
pushq %r13
pushq %r12
@ -2063,7 +2063,7 @@ _AES_GCM_decrypt:
pxor %xmm6, %xmm6
cmpl $12, %ebx
movl %ebx, %edx
jne L_AES_GCM_decrypt_iv_not_12
jne L_AES_GCM_decrypt_aesni_iv_not_12
# # Calculate values when IV is 12 bytes
# Set counter based on IV
movl $0x1000000, %ecx
@ -2103,7 +2103,7 @@ _AES_GCM_decrypt:
aesenc %xmm7, %xmm1
cmpl $11, %r10d
movdqa 160(%r15), %xmm7
jl L_AES_GCM_decrypt_calc_iv_12_last
jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1
movdqa 176(%r15), %xmm7
@ -2111,20 +2111,20 @@ _AES_GCM_decrypt:
aesenc %xmm7, %xmm1
cmpl $13, %r10d
movdqa 192(%r15), %xmm7
jl L_AES_GCM_decrypt_calc_iv_12_last
jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1
movdqa 208(%r15), %xmm7
aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1
movdqa 224(%r15), %xmm7
L_AES_GCM_decrypt_calc_iv_12_last:
L_AES_GCM_decrypt_aesni_calc_iv_12_last:
aesenclast %xmm7, %xmm5
aesenclast %xmm7, %xmm1
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
movdqu %xmm1, 144(%rsp)
jmp L_AES_GCM_decrypt_iv_done
L_AES_GCM_decrypt_iv_not_12:
jmp L_AES_GCM_decrypt_aesni_iv_done
L_AES_GCM_decrypt_aesni_iv_not_12:
# Calculate values when IV is not 12 bytes
# H = Encrypt X(=0)
movdqa (%r15), %xmm5
@ -2139,27 +2139,27 @@ L_AES_GCM_decrypt_iv_not_12:
aesenc 144(%r15), %xmm5
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5
aesenc 176(%r15), %xmm5
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5
aesenc 208(%r15), %xmm5
movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last:
L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast %xmm9, %xmm5
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
# Calc counter
# Initialization vector
cmpl $0x00, %edx
movq $0x00, %rcx
je L_AES_GCM_decrypt_calc_iv_done
je L_AES_GCM_decrypt_aesni_calc_iv_done
cmpl $16, %edx
jl L_AES_GCM_decrypt_calc_iv_lt16
jl L_AES_GCM_decrypt_aesni_calc_iv_lt16
andl $0xfffffff0, %edx
L_AES_GCM_decrypt_calc_iv_16_loop:
L_AES_GCM_decrypt_aesni_calc_iv_16_loop:
movdqu (%rax,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm4
@ -2219,22 +2219,22 @@ L_AES_GCM_decrypt_calc_iv_16_loop:
pxor %xmm2, %xmm4
addl $16, %ecx
cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_iv_16_loop
jl L_AES_GCM_decrypt_aesni_calc_iv_16_loop
movl %ebx, %edx
cmpl %edx, %ecx
je L_AES_GCM_decrypt_calc_iv_done
L_AES_GCM_decrypt_calc_iv_lt16:
je L_AES_GCM_decrypt_aesni_calc_iv_done
L_AES_GCM_decrypt_aesni_calc_iv_lt16:
subq $16, %rsp
pxor %xmm8, %xmm8
xorl %ebx, %ebx
movdqu %xmm8, (%rsp)
L_AES_GCM_decrypt_calc_iv_loop:
L_AES_GCM_decrypt_aesni_calc_iv_loop:
movzbl (%rax,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1)
incl %ecx
incl %ebx
cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_iv_loop
jl L_AES_GCM_decrypt_aesni_calc_iv_loop
movdqu (%rsp), %xmm8
addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -2293,7 +2293,7 @@ L_AES_GCM_decrypt_calc_iv_loop:
pxor %xmm1, %xmm2
pxor %xmm7, %xmm2
pxor %xmm2, %xmm4
L_AES_GCM_decrypt_calc_iv_done:
L_AES_GCM_decrypt_aesni_calc_iv_done:
# T = Encrypt counter
pxor %xmm0, %xmm0
shll $3, %edx
@ -2368,28 +2368,28 @@ L_AES_GCM_decrypt_calc_iv_done:
aesenc 144(%r15), %xmm8
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last:
L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast %xmm9, %xmm8
movdqu %xmm8, 144(%rsp)
L_AES_GCM_decrypt_iv_done:
L_AES_GCM_decrypt_aesni_iv_done:
# Additional authentication data
movl %r11d, %edx
cmpl $0x00, %edx
je L_AES_GCM_decrypt_calc_aad_done
je L_AES_GCM_decrypt_aesni_calc_aad_done
xorl %ecx, %ecx
cmpl $16, %edx
jl L_AES_GCM_decrypt_calc_aad_lt16
jl L_AES_GCM_decrypt_aesni_calc_aad_lt16
andl $0xfffffff0, %edx
L_AES_GCM_decrypt_calc_aad_16_loop:
L_AES_GCM_decrypt_aesni_calc_aad_16_loop:
movdqu (%r12,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm6
@ -2449,22 +2449,22 @@ L_AES_GCM_decrypt_calc_aad_16_loop:
pxor %xmm2, %xmm6
addl $16, %ecx
cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_aad_16_loop
jl L_AES_GCM_decrypt_aesni_calc_aad_16_loop
movl %r11d, %edx
cmpl %edx, %ecx
je L_AES_GCM_decrypt_calc_aad_done
L_AES_GCM_decrypt_calc_aad_lt16:
je L_AES_GCM_decrypt_aesni_calc_aad_done
L_AES_GCM_decrypt_aesni_calc_aad_lt16:
subq $16, %rsp
pxor %xmm8, %xmm8
xorl %ebx, %ebx
movdqu %xmm8, (%rsp)
L_AES_GCM_decrypt_calc_aad_loop:
L_AES_GCM_decrypt_aesni_calc_aad_loop:
movzbl (%r12,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1)
incl %ecx
incl %ebx
cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_aad_loop
jl L_AES_GCM_decrypt_aesni_calc_aad_loop
movdqu (%rsp), %xmm8
addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -2523,7 +2523,7 @@ L_AES_GCM_decrypt_calc_aad_loop:
pxor %xmm1, %xmm2
pxor %xmm7, %xmm2
pxor %xmm2, %xmm6
L_AES_GCM_decrypt_calc_aad_done:
L_AES_GCM_decrypt_aesni_calc_aad_done:
# Calculate counter and H
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
movdqa %xmm5, %xmm9
@ -2541,7 +2541,7 @@ L_AES_GCM_decrypt_calc_aad_done:
xorl %ebx, %ebx
cmpl $0x80, %r9d
movl %r9d, %r13d
jl L_AES_GCM_decrypt_done_128
jl L_AES_GCM_decrypt_aesni_done_128
andl $0xffffff80, %r13d
movdqa %xmm6, %xmm2
# H ^ 1
@ -2840,7 +2840,7 @@ L_AES_GCM_decrypt_calc_aad_done:
pxor %xmm8, %xmm14
pxor %xmm14, %xmm7
movdqu %xmm7, 112(%rsp)
L_AES_GCM_decrypt_ghash_128:
L_AES_GCM_decrypt_aesni_ghash_128:
leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8
@ -3108,7 +3108,7 @@ L_AES_GCM_decrypt_ghash_128:
pxor %xmm3, %xmm2
cmpl $11, %r10d
movdqa 160(%r15), %xmm7
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10
@ -3128,7 +3128,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc %xmm7, %xmm15
cmpl $13, %r10d
movdqa 192(%r15), %xmm7
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10
@ -3147,7 +3147,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc %xmm7, %xmm14
aesenc %xmm7, %xmm15
movdqa 224(%r15), %xmm7
L_AES_GCM_decrypt_aesenc_128_ghash_avx_done:
L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast %xmm7, %xmm8
aesenclast %xmm7, %xmm9
movdqu (%rcx), %xmm0
@ -3182,18 +3182,18 @@ L_AES_GCM_decrypt_aesenc_128_ghash_avx_done:
movdqu %xmm15, 112(%rdx)
addl $0x80, %ebx
cmpl %r13d, %ebx
jl L_AES_GCM_decrypt_ghash_128
jl L_AES_GCM_decrypt_aesni_ghash_128
movdqa %xmm2, %xmm6
movdqu (%rsp), %xmm5
L_AES_GCM_decrypt_done_128:
L_AES_GCM_decrypt_aesni_done_128:
movl %r9d, %edx
cmpl %edx, %ebx
jge L_AES_GCM_decrypt_done_dec
jge L_AES_GCM_decrypt_aesni_done_dec
movl %r9d, %r13d
andl $0xfffffff0, %r13d
cmpl %r13d, %ebx
jge L_AES_GCM_decrypt_last_block_done
L_AES_GCM_decrypt_last_block_start:
jge L_AES_GCM_decrypt_aesni_last_block_done
L_AES_GCM_decrypt_aesni_last_block_start:
leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx
movdqu (%rcx), %xmm1
@ -3242,28 +3242,28 @@ L_AES_GCM_decrypt_last_block_start:
pxor %xmm3, %xmm6
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_gfmul_last
jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_gfmul_last
jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_aesenc_gfmul_last:
L_AES_GCM_decrypt_aesni_aesenc_gfmul_last:
aesenclast %xmm9, %xmm8
movdqu (%rcx), %xmm9
pxor %xmm9, %xmm8
movdqu %xmm8, (%rdx)
addl $16, %ebx
cmpl %r13d, %ebx
jl L_AES_GCM_decrypt_last_block_start
L_AES_GCM_decrypt_last_block_done:
jl L_AES_GCM_decrypt_aesni_last_block_start
L_AES_GCM_decrypt_aesni_last_block_done:
movl %r9d, %ecx
movl %ecx, %edx
andl $15, %ecx
jz L_AES_GCM_decrypt_aesenc_last15_dec_avx_done
jz L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done
movdqu 128(%rsp), %xmm4
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
pxor (%r15), %xmm4
@ -3278,23 +3278,23 @@ L_AES_GCM_decrypt_last_block_done:
aesenc 144(%r15), %xmm4
cmpl $11, %r10d
movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc %xmm9, %xmm4
aesenc 176(%r15), %xmm4
cmpl $13, %r10d
movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc %xmm9, %xmm4
aesenc 208(%r15), %xmm4
movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last:
L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last:
aesenclast %xmm9, %xmm4
subq $32, %rsp
xorl %ecx, %ecx
movdqu %xmm4, (%rsp)
pxor %xmm0, %xmm0
movdqu %xmm0, 16(%rsp)
L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop:
movzbl (%rdi,%rbx,1), %r13d
movb %r13b, 16(%rsp,%rcx,1)
xorb (%rsp,%rcx,1), %r13b
@ -3302,7 +3302,7 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
incl %ebx
incl %ecx
cmpl %edx, %ebx
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop
jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop
movdqu 16(%rsp), %xmm4
addq $32, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm4
@ -3347,8 +3347,8 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
pxor %xmm13, %xmm14
pxor %xmm8, %xmm14
pxor %xmm14, %xmm6
L_AES_GCM_decrypt_aesenc_last15_dec_avx_done:
L_AES_GCM_decrypt_done_dec:
L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done:
L_AES_GCM_decrypt_aesni_done_dec:
movl %r9d, %edx
movl %r11d, %ecx
shlq $3, %rdx
@ -3400,24 +3400,24 @@ L_AES_GCM_decrypt_done_dec:
movdqu 144(%rsp), %xmm0
pxor %xmm6, %xmm0
cmpl $16, %r14d
je L_AES_GCM_decrypt_cmp_tag_16
je L_AES_GCM_decrypt_aesni_cmp_tag_16
subq $16, %rsp
xorq %rcx, %rcx
xorq %rbx, %rbx
movdqu %xmm0, (%rsp)
L_AES_GCM_decrypt_cmp_tag_loop:
L_AES_GCM_decrypt_aesni_cmp_tag_loop:
movzbl (%rsp,%rcx,1), %r13d
xorb (%r8,%rcx,1), %r13b
orb %r13b, %bl
incl %ecx
cmpl %r14d, %ecx
jne L_AES_GCM_decrypt_cmp_tag_loop
jne L_AES_GCM_decrypt_aesni_cmp_tag_loop
cmpb $0x00, %bl
sete %bl
addq $16, %rsp
xorq %rcx, %rcx
jmp L_AES_GCM_decrypt_cmp_tag_done
L_AES_GCM_decrypt_cmp_tag_16:
jmp L_AES_GCM_decrypt_aesni_cmp_tag_done
L_AES_GCM_decrypt_aesni_cmp_tag_16:
movdqu (%r8), %xmm1
pcmpeqb %xmm1, %xmm0
pmovmskb %xmm0, %rdx
@ -3425,7 +3425,7 @@ L_AES_GCM_decrypt_cmp_tag_16:
xorl %ebx, %ebx
cmpl $0xffff, %edx
sete %bl
L_AES_GCM_decrypt_cmp_tag_done:
L_AES_GCM_decrypt_aesni_cmp_tag_done:
movl %ebx, (%rbp)
addq $0xa8, %rsp
popq %rbp
@ -3436,7 +3436,7 @@ L_AES_GCM_decrypt_cmp_tag_done:
popq %r13
repz retq
#ifndef __APPLE__
.size AES_GCM_decrypt,.-AES_GCM_decrypt
.size AES_GCM_decrypt_aesni,.-AES_GCM_decrypt_aesni
#endif /* __APPLE__ */
#ifdef WOLFSSL_AESGCM_STREAM
#ifndef __APPLE__

276
wolfcrypt/src/aes_gcm_asm.asm
View File

@ -96,7 +96,7 @@ L_aes_gcm_mod2_128 QWORD 1, 13979173243358019584
ptr_L_aes_gcm_mod2_128 QWORD L_aes_gcm_mod2_128
_DATA ENDS
_text SEGMENT READONLY PARA
AES_GCM_encrypt PROC
AES_GCM_encrypt_aesni PROC
push r13
push rdi
push rsi
@ -130,7 +130,7 @@ AES_GCM_encrypt PROC
pxor xmm6, xmm6
cmp ebx, 12
mov edx, ebx
jne L_AES_GCM_encrypt_iv_not_12
jne L_AES_GCM_encrypt_aesni_iv_not_12
; # Calculate values when IV is 12 bytes
; Set counter based on IV
mov ecx, 16777216
@ -170,7 +170,7 @@ AES_GCM_encrypt PROC
aesenc xmm1, xmm7
cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_calc_iv_12_last
jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7
aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+176]
@ -178,20 +178,20 @@ AES_GCM_encrypt PROC
aesenc xmm1, xmm7
cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_calc_iv_12_last
jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7
aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+208]
aesenc xmm5, xmm7
aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_encrypt_calc_iv_12_last:
L_AES_GCM_encrypt_aesni_calc_iv_12_last:
aesenclast xmm5, xmm7
aesenclast xmm1, xmm7
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
movdqu [rsp+144], xmm1
jmp L_AES_GCM_encrypt_iv_done
L_AES_GCM_encrypt_iv_not_12:
jmp L_AES_GCM_encrypt_aesni_iv_done
L_AES_GCM_encrypt_aesni_iv_not_12:
; Calculate values when IV is not 12 bytes
; H = Encrypt X(=0)
movdqa xmm5, OWORD PTR [r15]
@ -206,27 +206,27 @@ L_AES_GCM_encrypt_iv_not_12:
aesenc xmm5, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9
aesenc xmm5, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9
aesenc xmm5, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast xmm5, xmm9
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
; Calc counter
; Initialization vector
cmp edx, 0
mov rcx, 0
je L_AES_GCM_encrypt_calc_iv_done
je L_AES_GCM_encrypt_aesni_calc_iv_done
cmp edx, 16
jl L_AES_GCM_encrypt_calc_iv_lt16
jl L_AES_GCM_encrypt_aesni_calc_iv_lt16
and edx, 4294967280
L_AES_GCM_encrypt_calc_iv_16_loop:
L_AES_GCM_encrypt_aesni_calc_iv_16_loop:
movdqu xmm8, [rax+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm4, xmm8
@ -286,22 +286,22 @@ L_AES_GCM_encrypt_calc_iv_16_loop:
pxor xmm4, xmm2
add ecx, 16
cmp ecx, edx
jl L_AES_GCM_encrypt_calc_iv_16_loop
jl L_AES_GCM_encrypt_aesni_calc_iv_16_loop
mov edx, ebx
cmp ecx, edx
je L_AES_GCM_encrypt_calc_iv_done
L_AES_GCM_encrypt_calc_iv_lt16:
je L_AES_GCM_encrypt_aesni_calc_iv_done
L_AES_GCM_encrypt_aesni_calc_iv_lt16:
sub rsp, 16
pxor xmm8, xmm8
xor ebx, ebx
movdqu [rsp], xmm8
L_AES_GCM_encrypt_calc_iv_loop:
L_AES_GCM_encrypt_aesni_calc_iv_loop:
movzx r13d, BYTE PTR [rax+rcx]
mov BYTE PTR [rsp+rbx], r13b
inc ecx
inc ebx
cmp ecx, edx
jl L_AES_GCM_encrypt_calc_iv_loop
jl L_AES_GCM_encrypt_aesni_calc_iv_loop
movdqu xmm8, [rsp]
add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -360,7 +360,7 @@ L_AES_GCM_encrypt_calc_iv_loop:
pxor xmm2, xmm1
pxor xmm2, xmm7
pxor xmm4, xmm2
L_AES_GCM_encrypt_calc_iv_done:
L_AES_GCM_encrypt_aesni_calc_iv_done:
; T = Encrypt counter
pxor xmm0, xmm0
shl edx, 3
@ -435,28 +435,28 @@ L_AES_GCM_encrypt_calc_iv_done:
aesenc xmm8, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast xmm8, xmm9
movdqu [rsp+144], xmm8
L_AES_GCM_encrypt_iv_done:
L_AES_GCM_encrypt_aesni_iv_done:
; Additional authentication data
mov edx, r11d
cmp edx, 0
je L_AES_GCM_encrypt_calc_aad_done
je L_AES_GCM_encrypt_aesni_calc_aad_done
xor ecx, ecx
cmp edx, 16
jl L_AES_GCM_encrypt_calc_aad_lt16
jl L_AES_GCM_encrypt_aesni_calc_aad_lt16
and edx, 4294967280
L_AES_GCM_encrypt_calc_aad_16_loop:
L_AES_GCM_encrypt_aesni_calc_aad_16_loop:
movdqu xmm8, [r12+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm6, xmm8
@ -516,22 +516,22 @@ L_AES_GCM_encrypt_calc_aad_16_loop:
pxor xmm6, xmm2
add ecx, 16
cmp ecx, edx
jl L_AES_GCM_encrypt_calc_aad_16_loop
jl L_AES_GCM_encrypt_aesni_calc_aad_16_loop
mov edx, r11d
cmp ecx, edx
je L_AES_GCM_encrypt_calc_aad_done
L_AES_GCM_encrypt_calc_aad_lt16:
je L_AES_GCM_encrypt_aesni_calc_aad_done
L_AES_GCM_encrypt_aesni_calc_aad_lt16:
sub rsp, 16
pxor xmm8, xmm8
xor ebx, ebx
movdqu [rsp], xmm8
L_AES_GCM_encrypt_calc_aad_loop:
L_AES_GCM_encrypt_aesni_calc_aad_loop:
movzx r13d, BYTE PTR [r12+rcx]
mov BYTE PTR [rsp+rbx], r13b
inc ecx
inc ebx
cmp ecx, edx
jl L_AES_GCM_encrypt_calc_aad_loop
jl L_AES_GCM_encrypt_aesni_calc_aad_loop
movdqu xmm8, [rsp]
add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -590,7 +590,7 @@ L_AES_GCM_encrypt_calc_aad_loop:
pxor xmm2, xmm1
pxor xmm2, xmm7
pxor xmm6, xmm2
L_AES_GCM_encrypt_calc_aad_done:
L_AES_GCM_encrypt_aesni_calc_aad_done:
; Calculate counter and H
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
movdqa xmm9, xmm5
@ -608,7 +608,7 @@ L_AES_GCM_encrypt_calc_aad_done:
xor rbx, rbx
cmp r9d, 128
mov r13d, r9d
jl L_AES_GCM_encrypt_done_128
jl L_AES_GCM_encrypt_aesni_done_128
and r13d, 4294967168
movdqa xmm2, xmm6
; H ^ 1
@ -1027,7 +1027,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc xmm15, xmm7
cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_enc_done
jl L_AES_GCM_encrypt_aesni_enc_done
aesenc xmm8, xmm7
aesenc xmm9, xmm7
aesenc xmm10, xmm7
@ -1047,7 +1047,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc xmm15, xmm7
cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_enc_done
jl L_AES_GCM_encrypt_aesni_enc_done
aesenc xmm8, xmm7
aesenc xmm9, xmm7
aesenc xmm10, xmm7
@ -1066,7 +1066,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc xmm14, xmm7
aesenc xmm15, xmm7
movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_encrypt_enc_done:
L_AES_GCM_encrypt_aesni_enc_done:
aesenclast xmm8, xmm7
aesenclast xmm9, xmm7
movdqu xmm0, [rdi]
@ -1101,9 +1101,9 @@ L_AES_GCM_encrypt_enc_done:
movdqu [rsi+112], xmm15
cmp r13d, 128
mov ebx, 128
jle L_AES_GCM_encrypt_end_128
jle L_AES_GCM_encrypt_aesni_end_128
; More 128 bytes of input
L_AES_GCM_encrypt_ghash_128:
L_AES_GCM_encrypt_aesni_ghash_128:
lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128]
@ -1371,7 +1371,7 @@ L_AES_GCM_encrypt_ghash_128:
pxor xmm2, xmm3
cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7
aesenc xmm9, xmm7
aesenc xmm10, xmm7
@ -1391,7 +1391,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc xmm15, xmm7
cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7
aesenc xmm9, xmm7
aesenc xmm10, xmm7
@ -1410,7 +1410,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc xmm14, xmm7
aesenc xmm15, xmm7
movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_128_ghash_avx_done:
L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast xmm8, xmm7
aesenclast xmm9, xmm7
movdqu xmm0, [rcx]
@ -1445,8 +1445,8 @@ L_AES_GCM_encrypt_aesenc_128_ghash_avx_done:
movdqu [rdx+112], xmm15
add ebx, 128
cmp ebx, r13d
jl L_AES_GCM_encrypt_ghash_128
L_AES_GCM_encrypt_end_128:
jl L_AES_GCM_encrypt_aesni_ghash_128
L_AES_GCM_encrypt_aesni_end_128:
movdqa xmm4, OWORD PTR L_aes_gcm_bswap_mask
pshufb xmm8, xmm4
pshufb xmm9, xmm4
@ -1633,14 +1633,14 @@ L_AES_GCM_encrypt_end_128:
pxor xmm2, xmm4
pxor xmm6, xmm2
movdqu xmm5, [rsp]
L_AES_GCM_encrypt_done_128:
L_AES_GCM_encrypt_aesni_done_128:
mov edx, r9d
cmp ebx, edx
jge L_AES_GCM_encrypt_done_enc
jge L_AES_GCM_encrypt_aesni_done_enc
mov r13d, r9d
and r13d, 4294967280
cmp ebx, r13d
jge L_AES_GCM_encrypt_last_block_done
jge L_AES_GCM_encrypt_aesni_last_block_done
lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128]
@ -1660,16 +1660,16 @@ L_AES_GCM_encrypt_done_128:
aesenc xmm8, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last:
aesenclast xmm8, xmm9
movdqu xmm9, [rcx]
pxor xmm8, xmm9
@ -1678,8 +1678,8 @@ L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last:
pxor xmm6, xmm8
add ebx, 16
cmp ebx, r13d
jge L_AES_GCM_encrypt_last_block_ghash
L_AES_GCM_encrypt_last_block_start:
jge L_AES_GCM_encrypt_aesni_last_block_ghash
L_AES_GCM_encrypt_aesni_last_block_start:
lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128]
@ -1724,16 +1724,16 @@ L_AES_GCM_encrypt_last_block_start:
pxor xmm6, xmm3
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_gfmul_last
jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_gfmul_last
jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_gfmul_last:
L_AES_GCM_encrypt_aesni_aesenc_gfmul_last:
aesenclast xmm8, xmm9
movdqu xmm9, [rcx]
pxor xmm8, xmm9
@ -1742,8 +1742,8 @@ L_AES_GCM_encrypt_aesenc_gfmul_last:
pxor xmm6, xmm8
add ebx, 16
cmp ebx, r13d
jl L_AES_GCM_encrypt_last_block_start
L_AES_GCM_encrypt_last_block_ghash:
jl L_AES_GCM_encrypt_aesni_last_block_start
L_AES_GCM_encrypt_aesni_last_block_ghash:
pshufd xmm9, xmm5, 78
pshufd xmm10, xmm6, 78
movdqa xmm11, xmm6
@ -1784,11 +1784,11 @@ L_AES_GCM_encrypt_last_block_ghash:
pxor xmm14, xmm13
pxor xmm14, xmm8
pxor xmm6, xmm14
L_AES_GCM_encrypt_last_block_done:
L_AES_GCM_encrypt_aesni_last_block_done:
mov ecx, r9d
mov edx, ecx
and ecx, 15
jz L_AES_GCM_encrypt_aesenc_last15_enc_avx_done
jz L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done
movdqu xmm4, [rsp+128]
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
pxor xmm4, [r15]
@ -1803,21 +1803,21 @@ L_AES_GCM_encrypt_last_block_done:
aesenc xmm4, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc xmm4, xmm9
aesenc xmm4, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc xmm4, xmm9
aesenc xmm4, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last:
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last:
aesenclast xmm4, xmm9
sub rsp, 16
xor ecx, ecx
movdqu [rsp], xmm4
L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop:
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop:
movzx r13d, BYTE PTR [rdi+rbx]
xor r13b, BYTE PTR [rsp+rcx]
mov BYTE PTR [rsi+rbx], r13b
@ -1825,16 +1825,16 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop:
inc ebx
inc ecx
cmp ebx, edx
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop
xor r13, r13
cmp ecx, 16
je L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc
L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop:
je L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop:
mov BYTE PTR [rsp+rcx], r13b
inc ecx
cmp ecx, 16
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop
L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc:
jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc:
movdqu xmm4, [rsp]
add rsp, 16
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_mask
@ -1879,8 +1879,8 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc:
pxor xmm14, xmm13
pxor xmm14, xmm8
pxor xmm6, xmm14
L_AES_GCM_encrypt_aesenc_last15_enc_avx_done:
L_AES_GCM_encrypt_done_enc:
L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done:
L_AES_GCM_encrypt_aesni_done_enc:
mov edx, r9d
mov ecx, r11d
shl rdx, 3
@ -1932,19 +1932,19 @@ L_AES_GCM_encrypt_done_enc:
movdqu xmm0, [rsp+144]
pxor xmm0, xmm6
cmp r14d, 16
je L_AES_GCM_encrypt_store_tag_16
je L_AES_GCM_encrypt_aesni_store_tag_16
xor rcx, rcx
movdqu [rsp], xmm0
L_AES_GCM_encrypt_store_tag_loop:
L_AES_GCM_encrypt_aesni_store_tag_loop:
movzx r13d, BYTE PTR [rsp+rcx]
mov BYTE PTR [r8+rcx], r13b
inc ecx
cmp ecx, r14d
jne L_AES_GCM_encrypt_store_tag_loop
jmp L_AES_GCM_encrypt_store_tag_done
L_AES_GCM_encrypt_store_tag_16:
jne L_AES_GCM_encrypt_aesni_store_tag_loop
jmp L_AES_GCM_encrypt_aesni_store_tag_done
L_AES_GCM_encrypt_aesni_store_tag_16:
movdqu [r8], xmm0
L_AES_GCM_encrypt_store_tag_done:
L_AES_GCM_encrypt_aesni_store_tag_done:
movdqu xmm6, [rsp+160]
movdqu xmm7, [rsp+176]
movdqu xmm8, [rsp+192]
@ -1964,10 +1964,10 @@ L_AES_GCM_encrypt_store_tag_done:
pop rdi
pop r13
ret
AES_GCM_encrypt ENDP
AES_GCM_encrypt_aesni ENDP
_text ENDS
_text SEGMENT READONLY PARA
AES_GCM_decrypt PROC
AES_GCM_decrypt_aesni PROC
push r13
push rdi
push rsi
@ -2003,7 +2003,7 @@ AES_GCM_decrypt PROC
pxor xmm6, xmm6
cmp ebx, 12
mov edx, ebx
jne L_AES_GCM_decrypt_iv_not_12
jne L_AES_GCM_decrypt_aesni_iv_not_12
; # Calculate values when IV is 12 bytes
; Set counter based on IV
mov ecx, 16777216
@ -2043,7 +2043,7 @@ AES_GCM_decrypt PROC
aesenc xmm1, xmm7
cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_calc_iv_12_last
jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7
aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+176]
@ -2051,20 +2051,20 @@ AES_GCM_decrypt PROC
aesenc xmm1, xmm7
cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_calc_iv_12_last
jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7
aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+208]
aesenc xmm5, xmm7
aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_decrypt_calc_iv_12_last:
L_AES_GCM_decrypt_aesni_calc_iv_12_last:
aesenclast xmm5, xmm7
aesenclast xmm1, xmm7
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
movdqu [rsp+144], xmm1
jmp L_AES_GCM_decrypt_iv_done
L_AES_GCM_decrypt_iv_not_12:
jmp L_AES_GCM_decrypt_aesni_iv_done
L_AES_GCM_decrypt_aesni_iv_not_12:
; Calculate values when IV is not 12 bytes
; H = Encrypt X(=0)
movdqa xmm5, OWORD PTR [r15]
@ -2079,27 +2079,27 @@ L_AES_GCM_decrypt_iv_not_12:
aesenc xmm5, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9
aesenc xmm5, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9
aesenc xmm5, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last:
L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast xmm5, xmm9
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
; Calc counter
; Initialization vector
cmp edx, 0
mov rcx, 0
je L_AES_GCM_decrypt_calc_iv_done
je L_AES_GCM_decrypt_aesni_calc_iv_done
cmp edx, 16
jl L_AES_GCM_decrypt_calc_iv_lt16
jl L_AES_GCM_decrypt_aesni_calc_iv_lt16
and edx, 4294967280
L_AES_GCM_decrypt_calc_iv_16_loop:
L_AES_GCM_decrypt_aesni_calc_iv_16_loop:
movdqu xmm8, [rax+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm4, xmm8
@ -2159,22 +2159,22 @@ L_AES_GCM_decrypt_calc_iv_16_loop:
pxor xmm4, xmm2
add ecx, 16
cmp ecx, edx
jl L_AES_GCM_decrypt_calc_iv_16_loop
jl L_AES_GCM_decrypt_aesni_calc_iv_16_loop
mov edx, ebx
cmp ecx, edx
je L_AES_GCM_decrypt_calc_iv_done
L_AES_GCM_decrypt_calc_iv_lt16:
je L_AES_GCM_decrypt_aesni_calc_iv_done
L_AES_GCM_decrypt_aesni_calc_iv_lt16:
sub rsp, 16
pxor xmm8, xmm8
xor ebx, ebx
movdqu [rsp], xmm8
L_AES_GCM_decrypt_calc_iv_loop:
L_AES_GCM_decrypt_aesni_calc_iv_loop:
movzx r13d, BYTE PTR [rax+rcx]
mov BYTE PTR [rsp+rbx], r13b
inc ecx
inc ebx
cmp ecx, edx
jl L_AES_GCM_decrypt_calc_iv_loop
jl L_AES_GCM_decrypt_aesni_calc_iv_loop
movdqu xmm8, [rsp]
add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -2233,7 +2233,7 @@ L_AES_GCM_decrypt_calc_iv_loop:
pxor xmm2, xmm1
pxor xmm2, xmm7
pxor xmm4, xmm2
L_AES_GCM_decrypt_calc_iv_done:
L_AES_GCM_decrypt_aesni_calc_iv_done:
; T = Encrypt counter
pxor xmm0, xmm0
shl edx, 3
@ -2308,28 +2308,28 @@ L_AES_GCM_decrypt_calc_iv_done:
aesenc xmm8, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last:
L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast xmm8, xmm9
movdqu [rsp+144], xmm8
L_AES_GCM_decrypt_iv_done:
L_AES_GCM_decrypt_aesni_iv_done:
; Additional authentication data
mov edx, r11d
cmp edx, 0
je L_AES_GCM_decrypt_calc_aad_done
je L_AES_GCM_decrypt_aesni_calc_aad_done
xor ecx, ecx
cmp edx, 16
jl L_AES_GCM_decrypt_calc_aad_lt16
jl L_AES_GCM_decrypt_aesni_calc_aad_lt16
and edx, 4294967280
L_AES_GCM_decrypt_calc_aad_16_loop:
L_AES_GCM_decrypt_aesni_calc_aad_16_loop:
movdqu xmm8, [r12+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm6, xmm8
@ -2389,22 +2389,22 @@ L_AES_GCM_decrypt_calc_aad_16_loop:
pxor xmm6, xmm2
add ecx, 16
cmp ecx, edx
jl L_AES_GCM_decrypt_calc_aad_16_loop
jl L_AES_GCM_decrypt_aesni_calc_aad_16_loop
mov edx, r11d
cmp ecx, edx
je L_AES_GCM_decrypt_calc_aad_done
L_AES_GCM_decrypt_calc_aad_lt16:
je L_AES_GCM_decrypt_aesni_calc_aad_done
L_AES_GCM_decrypt_aesni_calc_aad_lt16:
sub rsp, 16
pxor xmm8, xmm8
xor ebx, ebx
movdqu [rsp], xmm8
L_AES_GCM_decrypt_calc_aad_loop:
L_AES_GCM_decrypt_aesni_calc_aad_loop:
movzx r13d, BYTE PTR [r12+rcx]
mov BYTE PTR [rsp+rbx], r13b
inc ecx
inc ebx
cmp ecx, edx
jl L_AES_GCM_decrypt_calc_aad_loop
jl L_AES_GCM_decrypt_aesni_calc_aad_loop
movdqu xmm8, [rsp]
add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -2463,7 +2463,7 @@ L_AES_GCM_decrypt_calc_aad_loop:
pxor xmm2, xmm1
pxor xmm2, xmm7
pxor xmm6, xmm2
L_AES_GCM_decrypt_calc_aad_done:
L_AES_GCM_decrypt_aesni_calc_aad_done:
; Calculate counter and H
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
movdqa xmm9, xmm5
@ -2481,7 +2481,7 @@ L_AES_GCM_decrypt_calc_aad_done:
xor ebx, ebx
cmp r9d, 128
mov r13d, r9d
jl L_AES_GCM_decrypt_done_128
jl L_AES_GCM_decrypt_aesni_done_128
and r13d, 4294967168
movdqa xmm2, xmm6
; H ^ 1
@ -2780,7 +2780,7 @@ L_AES_GCM_decrypt_calc_aad_done:
pxor xmm14, xmm8
pxor xmm7, xmm14
movdqu [rsp+112], xmm7
L_AES_GCM_decrypt_ghash_128:
L_AES_GCM_decrypt_aesni_ghash_128:
lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128]
@ -3048,7 +3048,7 @@ L_AES_GCM_decrypt_ghash_128:
pxor xmm2, xmm3
cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7
aesenc xmm9, xmm7
aesenc xmm10, xmm7
@ -3068,7 +3068,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc xmm15, xmm7
cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done
jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7
aesenc xmm9, xmm7
aesenc xmm10, xmm7
@ -3087,7 +3087,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc xmm14, xmm7
aesenc xmm15, xmm7
movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_decrypt_aesenc_128_ghash_avx_done:
L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast xmm8, xmm7
aesenclast xmm9, xmm7
movdqu xmm0, [rcx]
@ -3122,18 +3122,18 @@ L_AES_GCM_decrypt_aesenc_128_ghash_avx_done:
movdqu [rdx+112], xmm15
add ebx, 128
cmp ebx, r13d
jl L_AES_GCM_decrypt_ghash_128
jl L_AES_GCM_decrypt_aesni_ghash_128
movdqa xmm6, xmm2
movdqu xmm5, [rsp]
L_AES_GCM_decrypt_done_128:
L_AES_GCM_decrypt_aesni_done_128:
mov edx, r9d
cmp ebx, edx
jge L_AES_GCM_decrypt_done_dec
jge L_AES_GCM_decrypt_aesni_done_dec
mov r13d, r9d
and r13d, 4294967280
cmp ebx, r13d
jge L_AES_GCM_decrypt_last_block_done
L_AES_GCM_decrypt_last_block_start:
jge L_AES_GCM_decrypt_aesni_last_block_done
L_AES_GCM_decrypt_aesni_last_block_start:
lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm1, [rcx]
@ -3182,28 +3182,28 @@ L_AES_GCM_decrypt_last_block_start:
pxor xmm6, xmm3
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_aesenc_gfmul_last
jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_aesenc_gfmul_last
jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9
aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_aesenc_gfmul_last:
L_AES_GCM_decrypt_aesni_aesenc_gfmul_last:
aesenclast xmm8, xmm9
movdqu xmm9, [rcx]
pxor xmm8, xmm9
movdqu [rdx], xmm8
add ebx, 16
cmp ebx, r13d
jl L_AES_GCM_decrypt_last_block_start
L_AES_GCM_decrypt_last_block_done:
jl L_AES_GCM_decrypt_aesni_last_block_start
L_AES_GCM_decrypt_aesni_last_block_done:
mov ecx, r9d
mov edx, ecx
and ecx, 15
jz L_AES_GCM_decrypt_aesenc_last15_dec_avx_done
jz L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done
movdqu xmm4, [rsp+128]
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
pxor xmm4, [r15]
@ -3218,23 +3218,23 @@ L_AES_GCM_decrypt_last_block_done:
aesenc xmm4, [r15+144]
cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc xmm4, xmm9
aesenc xmm4, [r15+176]
cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last
jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc xmm4, xmm9
aesenc xmm4, [r15+208]
movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last:
L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last:
aesenclast xmm4, xmm9
sub rsp, 32
xor ecx, ecx
movdqu [rsp], xmm4
pxor xmm0, xmm0
movdqu [rsp+16], xmm0
L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop:
movzx r13d, BYTE PTR [rdi+rbx]
mov BYTE PTR [rsp+rcx+16], r13b
xor r13b, BYTE PTR [rsp+rcx]
@ -3242,7 +3242,7 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
inc ebx
inc ecx
cmp ebx, edx
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop
jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop
movdqu xmm4, [rsp+16]
add rsp, 32
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_mask
@ -3287,8 +3287,8 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
pxor xmm14, xmm13
pxor xmm14, xmm8
pxor xmm6, xmm14
L_AES_GCM_decrypt_aesenc_last15_dec_avx_done:
L_AES_GCM_decrypt_done_dec:
L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done:
L_AES_GCM_decrypt_aesni_done_dec:
mov edx, r9d
mov ecx, r11d
shl rdx, 3
@ -3340,24 +3340,24 @@ L_AES_GCM_decrypt_done_dec:
movdqu xmm0, [rsp+144]
pxor xmm0, xmm6
cmp r14d, 16
je L_AES_GCM_decrypt_cmp_tag_16
je L_AES_GCM_decrypt_aesni_cmp_tag_16
sub rsp, 16
xor rcx, rcx
xor rbx, rbx
movdqu [rsp], xmm0
L_AES_GCM_decrypt_cmp_tag_loop:
L_AES_GCM_decrypt_aesni_cmp_tag_loop:
movzx r13d, BYTE PTR [rsp+rcx]
xor r13b, BYTE PTR [r8+rcx]
or bl, r13b
inc ecx
cmp ecx, r14d
jne L_AES_GCM_decrypt_cmp_tag_loop
jne L_AES_GCM_decrypt_aesni_cmp_tag_loop
cmp rbx, 0
sete bl
add rsp, 16
xor rcx, rcx
jmp L_AES_GCM_decrypt_cmp_tag_done
L_AES_GCM_decrypt_cmp_tag_16:
jmp L_AES_GCM_decrypt_aesni_cmp_tag_done
L_AES_GCM_decrypt_aesni_cmp_tag_16:
movdqu xmm1, [r8]
pcmpeqb xmm0, xmm1
pmovmskb rdx, xmm0
@ -3365,7 +3365,7 @@ L_AES_GCM_decrypt_cmp_tag_16:
xor ebx, ebx
cmp edx, 65535
sete bl
L_AES_GCM_decrypt_cmp_tag_done:
L_AES_GCM_decrypt_aesni_cmp_tag_done:
mov DWORD PTR [rbp], ebx
movdqu xmm6, [rsp+168]
movdqu xmm7, [rsp+184]
@ -3387,7 +3387,7 @@ L_AES_GCM_decrypt_cmp_tag_done:
pop rdi
pop r13
ret
AES_GCM_decrypt ENDP
AES_GCM_decrypt_aesni ENDP
_text ENDS
_text SEGMENT READONLY PARA
AES_GCM_init_aesni PROC

140
wolfcrypt/src/aes_xts_asm.S
View File

@ -56,15 +56,15 @@ L_aes_xts_gc_xts:
.long 0x87,0x1,0x1,0x1
#ifndef __APPLE__
.text
.globl AES_XTS_encrypt
.type AES_XTS_encrypt,@function
.globl AES_XTS_encrypt_aesni
.type AES_XTS_encrypt_aesni,@function
.align 16
AES_XTS_encrypt:
AES_XTS_encrypt_aesni:
#else
.section __TEXT,__text
.globl _AES_XTS_encrypt
.globl _AES_XTS_encrypt_aesni
.p2align 4
_AES_XTS_encrypt:
_AES_XTS_encrypt_aesni:
#endif /* __APPLE__ */
pushq %r12
pushq %r13
@ -96,25 +96,25 @@ _AES_XTS_encrypt:
aesenc %xmm5, %xmm0
cmpl $11, %r10d
movdqu 160(%r9), %xmm5
jl L_AES_XTS_encrypt_tweak_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0
movdqu 176(%r9), %xmm6
aesenc %xmm6, %xmm0
cmpl $13, %r10d
movdqu 192(%r9), %xmm5
jl L_AES_XTS_encrypt_tweak_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0
movdqu 208(%r9), %xmm6
aesenc %xmm6, %xmm0
movdqu 224(%r9), %xmm5
L_AES_XTS_encrypt_tweak_aes_enc_block_last:
L_AES_XTS_encrypt_aesni_tweak_aes_enc_block_last:
aesenclast %xmm5, %xmm0
xorl %r13d, %r13d
cmpl $0x40, %eax
movl %eax, %r11d
jl L_AES_XTS_encrypt_done_64
jl L_AES_XTS_encrypt_aesni_done_64
andl $0xffffffc0, %r11d
L_AES_XTS_encrypt_enc_64:
L_AES_XTS_encrypt_aesni_enc_64:
# 64 bytes of input
# aes_enc_64
leaq (%rdi,%r13,1), %rcx
@ -201,7 +201,7 @@ L_AES_XTS_encrypt_enc_64:
aesenc %xmm4, %xmm11
cmpl $11, %r10d
movdqu 160(%r8), %xmm4
jl L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_aes_enc_64_aes_enc_block_last
aesenc %xmm4, %xmm8
aesenc %xmm4, %xmm9
aesenc %xmm4, %xmm10
@ -213,7 +213,7 @@ L_AES_XTS_encrypt_enc_64:
aesenc %xmm4, %xmm11
cmpl $13, %r10d
movdqu 192(%r8), %xmm4
jl L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_aes_enc_64_aes_enc_block_last
aesenc %xmm4, %xmm8
aesenc %xmm4, %xmm9
aesenc %xmm4, %xmm10
@ -224,7 +224,7 @@ L_AES_XTS_encrypt_enc_64:
aesenc %xmm4, %xmm10
aesenc %xmm4, %xmm11
movdqu 224(%r8), %xmm4
L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last:
L_AES_XTS_encrypt_aesni_aes_enc_64_aes_enc_block_last:
aesenclast %xmm4, %xmm8
aesenclast %xmm4, %xmm9
aesenclast %xmm4, %xmm10
@ -246,18 +246,18 @@ L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last:
pxor %xmm4, %xmm0
addl $0x40, %r13d
cmpl %r11d, %r13d
jl L_AES_XTS_encrypt_enc_64
L_AES_XTS_encrypt_done_64:
jl L_AES_XTS_encrypt_aesni_enc_64
L_AES_XTS_encrypt_aesni_done_64:
cmpl %eax, %r13d
movl %eax, %r11d
je L_AES_XTS_encrypt_done_enc
je L_AES_XTS_encrypt_aesni_done_enc
subl %r13d, %r11d
cmpl $16, %r11d
movl %eax, %r11d
jl L_AES_XTS_encrypt_last_15
jl L_AES_XTS_encrypt_aesni_last_15
andl $0xfffffff0, %r11d
# 16 bytes of input
L_AES_XTS_encrypt_enc_16:
L_AES_XTS_encrypt_aesni_enc_16:
leaq (%rdi,%r13,1), %rcx
movdqu (%rcx), %xmm8
pxor %xmm0, %xmm8
@ -283,18 +283,18 @@ L_AES_XTS_encrypt_enc_16:
aesenc %xmm5, %xmm8
cmpl $11, %r10d
movdqu 160(%r8), %xmm5
jl L_AES_XTS_encrypt_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_aes_enc_block_last
aesenc %xmm5, %xmm8
movdqu 176(%r8), %xmm6
aesenc %xmm6, %xmm8
cmpl $13, %r10d
movdqu 192(%r8), %xmm5
jl L_AES_XTS_encrypt_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_aes_enc_block_last
aesenc %xmm5, %xmm8
movdqu 208(%r8), %xmm6
aesenc %xmm6, %xmm8
movdqu 224(%r8), %xmm5
L_AES_XTS_encrypt_aes_enc_block_last:
L_AES_XTS_encrypt_aesni_aes_enc_block_last:
aesenclast %xmm5, %xmm8
pxor %xmm0, %xmm8
leaq (%rsi,%r13,1), %rcx
@ -307,17 +307,17 @@ L_AES_XTS_encrypt_aes_enc_block_last:
pxor %xmm4, %xmm0
addl $16, %r13d
cmpl %r11d, %r13d
jl L_AES_XTS_encrypt_enc_16
jl L_AES_XTS_encrypt_aesni_enc_16
cmpl %eax, %r13d
je L_AES_XTS_encrypt_done_enc
L_AES_XTS_encrypt_last_15:
je L_AES_XTS_encrypt_aesni_done_enc
L_AES_XTS_encrypt_aesni_last_15:
subq $16, %r13
leaq (%rsi,%r13,1), %rcx
movdqu (%rcx), %xmm8
addq $16, %r13
movdqu %xmm8, (%rsp)
xorq %rdx, %rdx
L_AES_XTS_encrypt_last_15_byte_loop:
L_AES_XTS_encrypt_aesni_last_15_byte_loop:
movb (%rsp,%rdx,1), %r11b
movb (%rdi,%r13,1), %cl
movb %r11b, (%rsi,%r13,1)
@ -325,7 +325,7 @@ L_AES_XTS_encrypt_last_15_byte_loop:
incl %r13d
incl %edx
cmpl %eax, %r13d
jl L_AES_XTS_encrypt_last_15_byte_loop
jl L_AES_XTS_encrypt_aesni_last_15_byte_loop
subq %rdx, %r13
movdqu (%rsp), %xmm8
subq $16, %r13
@ -352,41 +352,41 @@ L_AES_XTS_encrypt_last_15_byte_loop:
aesenc %xmm5, %xmm8
cmpl $11, %r10d
movdqu 160(%r8), %xmm5
jl L_AES_XTS_encrypt_last_15_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_last_15_aes_enc_block_last
aesenc %xmm5, %xmm8
movdqu 176(%r8), %xmm6
aesenc %xmm6, %xmm8
cmpl $13, %r10d
movdqu 192(%r8), %xmm5
jl L_AES_XTS_encrypt_last_15_aes_enc_block_last
jl L_AES_XTS_encrypt_aesni_last_15_aes_enc_block_last
aesenc %xmm5, %xmm8
movdqu 208(%r8), %xmm6
aesenc %xmm6, %xmm8
movdqu 224(%r8), %xmm5
L_AES_XTS_encrypt_last_15_aes_enc_block_last:
L_AES_XTS_encrypt_aesni_last_15_aes_enc_block_last:
aesenclast %xmm5, %xmm8
pxor %xmm0, %xmm8
leaq (%rsi,%r13,1), %rcx
movdqu %xmm8, (%rcx)
L_AES_XTS_encrypt_done_enc:
L_AES_XTS_encrypt_aesni_done_enc:
addq $0x40, %rsp
popq %r13
popq %r12
repz retq
#ifndef __APPLE__
.size AES_XTS_encrypt,.-AES_XTS_encrypt
.size AES_XTS_encrypt_aesni,.-AES_XTS_encrypt_aesni
#endif /* __APPLE__ */
#ifndef __APPLE__
.text
.globl AES_XTS_decrypt
.type AES_XTS_decrypt,@function
.globl AES_XTS_decrypt_aesni
.type AES_XTS_decrypt_aesni,@function
.align 16
AES_XTS_decrypt:
AES_XTS_decrypt_aesni:
#else
.section __TEXT,__text
.globl _AES_XTS_decrypt
.globl _AES_XTS_decrypt_aesni
.p2align 4
_AES_XTS_decrypt:
_AES_XTS_decrypt_aesni:
#endif /* __APPLE__ */
pushq %r12
pushq %r13
@ -418,32 +418,32 @@ _AES_XTS_decrypt:
aesenc %xmm5, %xmm0
cmpl $11, %r10d
movdqu 160(%r9), %xmm5
jl L_AES_XTS_decrypt_tweak_aes_enc_block_last
jl L_AES_XTS_decrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0
movdqu 176(%r9), %xmm6
aesenc %xmm6, %xmm0
cmpl $13, %r10d
movdqu 192(%r9), %xmm5
jl L_AES_XTS_decrypt_tweak_aes_enc_block_last
jl L_AES_XTS_decrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0
movdqu 208(%r9), %xmm6
aesenc %xmm6, %xmm0
movdqu 224(%r9), %xmm5
L_AES_XTS_decrypt_tweak_aes_enc_block_last:
L_AES_XTS_decrypt_aesni_tweak_aes_enc_block_last:
aesenclast %xmm5, %xmm0
xorl %r13d, %r13d
movl %eax, %r11d
andl $0xfffffff0, %r11d
cmpl %eax, %r11d
je L_AES_XTS_decrypt_mul16_64
je L_AES_XTS_decrypt_aesni_mul16_64
subl $16, %r11d
cmpl $16, %r11d
jl L_AES_XTS_decrypt_last_31_start
L_AES_XTS_decrypt_mul16_64:
jl L_AES_XTS_decrypt_aesni_last_31_start
L_AES_XTS_decrypt_aesni_mul16_64:
cmpl $0x40, %r11d
jl L_AES_XTS_decrypt_done_64
jl L_AES_XTS_decrypt_aesni_done_64
andl $0xffffffc0, %r11d
L_AES_XTS_decrypt_dec_64:
L_AES_XTS_decrypt_aesni_dec_64:
# 64 bytes of input
# aes_dec_64
leaq (%rdi,%r13,1), %rcx
@ -530,7 +530,7 @@ L_AES_XTS_decrypt_dec_64:
aesdec %xmm4, %xmm11
cmpl $11, %r10d
movdqu 160(%r8), %xmm4
jl L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_aes_dec_64_aes_dec_block_last
aesdec %xmm4, %xmm8
aesdec %xmm4, %xmm9
aesdec %xmm4, %xmm10
@ -542,7 +542,7 @@ L_AES_XTS_decrypt_dec_64:
aesdec %xmm4, %xmm11
cmpl $13, %r10d
movdqu 192(%r8), %xmm4
jl L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_aes_dec_64_aes_dec_block_last
aesdec %xmm4, %xmm8
aesdec %xmm4, %xmm9
aesdec %xmm4, %xmm10
@ -553,7 +553,7 @@ L_AES_XTS_decrypt_dec_64:
aesdec %xmm4, %xmm10
aesdec %xmm4, %xmm11
movdqu 224(%r8), %xmm4
L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last:
L_AES_XTS_decrypt_aesni_aes_dec_64_aes_dec_block_last:
aesdeclast %xmm4, %xmm8
aesdeclast %xmm4, %xmm9
aesdeclast %xmm4, %xmm10
@ -575,21 +575,21 @@ L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last:
pxor %xmm4, %xmm0
addl $0x40, %r13d
cmpl %r11d, %r13d
jl L_AES_XTS_decrypt_dec_64
L_AES_XTS_decrypt_done_64:
jl L_AES_XTS_decrypt_aesni_dec_64
L_AES_XTS_decrypt_aesni_done_64:
cmpl %eax, %r13d
movl %eax, %r11d
je L_AES_XTS_decrypt_done_dec
je L_AES_XTS_decrypt_aesni_done_dec
andl $0xfffffff0, %r11d
cmpl %eax, %r11d
je L_AES_XTS_decrypt_mul16
je L_AES_XTS_decrypt_aesni_mul16
subl $16, %r11d
subl %r13d, %r11d
cmpl $16, %r11d
jl L_AES_XTS_decrypt_last_31_start
jl L_AES_XTS_decrypt_aesni_last_31_start
addl %r13d, %r11d
L_AES_XTS_decrypt_mul16:
L_AES_XTS_decrypt_dec_16:
L_AES_XTS_decrypt_aesni_mul16:
L_AES_XTS_decrypt_aesni_dec_16:
# 16 bytes of input
leaq (%rdi,%r13,1), %rcx
movdqu (%rcx), %xmm8
@ -616,18 +616,18 @@ L_AES_XTS_decrypt_dec_16:
aesdec %xmm5, %xmm8
cmpl $11, %r10d
movdqu 160(%r8), %xmm5
jl L_AES_XTS_decrypt_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_aes_dec_block_last
aesdec %xmm5, %xmm8
movdqu 176(%r8), %xmm6
aesdec %xmm6, %xmm8
cmpl $13, %r10d
movdqu 192(%r8), %xmm5
jl L_AES_XTS_decrypt_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_aes_dec_block_last
aesdec %xmm5, %xmm8
movdqu 208(%r8), %xmm6
aesdec %xmm6, %xmm8
movdqu 224(%r8), %xmm5
L_AES_XTS_decrypt_aes_dec_block_last:
L_AES_XTS_decrypt_aesni_aes_dec_block_last:
aesdeclast %xmm5, %xmm8
pxor %xmm0, %xmm8
leaq (%rsi,%r13,1), %rcx
@ -640,10 +640,10 @@ L_AES_XTS_decrypt_aes_dec_block_last:
pxor %xmm4, %xmm0
addl $16, %r13d
cmpl %r11d, %r13d
jl L_AES_XTS_decrypt_dec_16
jl L_AES_XTS_decrypt_aesni_dec_16
cmpl %eax, %r13d
je L_AES_XTS_decrypt_done_dec
L_AES_XTS_decrypt_last_31_start:
je L_AES_XTS_decrypt_aesni_done_dec
L_AES_XTS_decrypt_aesni_last_31_start:
movdqa %xmm0, %xmm4
movdqa %xmm0, %xmm7
psrad $31, %xmm4
@ -676,24 +676,24 @@ L_AES_XTS_decrypt_last_31_start:
aesdec %xmm5, %xmm8
cmpl $11, %r10d
movdqu 160(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_last_31_aes_dec_block_last
aesdec %xmm5, %xmm8
movdqu 176(%r8), %xmm6
aesdec %xmm6, %xmm8
cmpl $13, %r10d
movdqu 192(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_last_31_aes_dec_block_last
aesdec %xmm5, %xmm8
movdqu 208(%r8), %xmm6
aesdec %xmm6, %xmm8
movdqu 224(%r8), %xmm5
L_AES_XTS_decrypt_last_31_aes_dec_block_last:
L_AES_XTS_decrypt_aesni_last_31_aes_dec_block_last:
aesdeclast %xmm5, %xmm8
pxor %xmm7, %xmm8
movdqu %xmm8, (%rsp)
addq $16, %r13
xorq %rdx, %rdx
L_AES_XTS_decrypt_last_31_byte_loop:
L_AES_XTS_decrypt_aesni_last_31_byte_loop:
movb (%rsp,%rdx,1), %r11b
movb (%rdi,%r13,1), %cl
movb %r11b, (%rsi,%r13,1)
@ -701,7 +701,7 @@ L_AES_XTS_decrypt_last_31_byte_loop:
incl %r13d
incl %edx
cmpl %eax, %r13d
jl L_AES_XTS_decrypt_last_31_byte_loop
jl L_AES_XTS_decrypt_aesni_last_31_byte_loop
subq %rdx, %r13
movdqu (%rsp), %xmm8
pxor %xmm0, %xmm8
@ -727,30 +727,30 @@ L_AES_XTS_decrypt_last_31_byte_loop:
aesdec %xmm5, %xmm8
cmpl $11, %r10d
movdqu 160(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_2_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_last_31_2_aes_dec_block_last
aesdec %xmm5, %xmm8
movdqu 176(%r8), %xmm6
aesdec %xmm6, %xmm8
cmpl $13, %r10d
movdqu 192(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_2_aes_dec_block_last
jl L_AES_XTS_decrypt_aesni_last_31_2_aes_dec_block_last
aesdec %xmm5, %xmm8
movdqu 208(%r8), %xmm6
aesdec %xmm6, %xmm8
movdqu 224(%r8), %xmm5
L_AES_XTS_decrypt_last_31_2_aes_dec_block_last:
L_AES_XTS_decrypt_aesni_last_31_2_aes_dec_block_last:
aesdeclast %xmm5, %xmm8
pxor %xmm0, %xmm8
subq $16, %r13
leaq (%rsi,%r13,1), %rcx
movdqu %xmm8, (%rcx)
L_AES_XTS_decrypt_done_dec:
L_AES_XTS_decrypt_aesni_done_dec:
addq $16, %rsp
popq %r13
popq %r12
repz retq
#ifndef __APPLE__
.size AES_XTS_decrypt,.-AES_XTS_decrypt
.size AES_XTS_decrypt_aesni,.-AES_XTS_decrypt_aesni
#endif /* __APPLE__ */
#ifdef HAVE_INTEL_AVX1
#ifndef __APPLE__

5
wolfcrypt/src/asn.c
View File

@ -30979,6 +30979,9 @@ int wc_SetSubjectKeyId(Cert *cert, const char* file)
#ifdef HAVE_ECC
wc_ecc_free(eckey);
XFREE(eckey, cert->heap, DYNAMIC_TYPE_ECC);
#endif
#if defined(NO_RSA) && !defined(HAVE_ECC)
(void)idx;
#endif
return ret;
}
@ -32191,7 +32194,7 @@ int DecodeECC_DSA_Sig_Ex(const byte* sig, word32 sigLen, mp_int* r, mp_int* s,
#ifdef WOLFSSL_ASN_TEMPLATE
#ifdef WOLFSSL_CUSTOM_CURVES
#if defined(HAVE_ECC) && defined(WOLFSSL_CUSTOM_CURVES)
/* Convert data to hex string.
*
* Big-endian byte array is converted to big-endian hexadecimal string.

4
wolfcrypt/src/evp.c
View File

@ -4672,7 +4672,7 @@ int wolfSSL_EVP_read_pw_string(char* buf, int bufSz, const char* banner, int v)
}
#endif /* WOLFSSL_APACHE_HTTPD */
#if !defined(NO_PWDBASED) && !defined(NO_SHA)
#if !defined(NO_PWDBASED) && !defined(NO_SHA) && !defined(NO_HMAC)
int wolfSSL_PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
const unsigned char *salt,
int saltlen, int iter,
@ -4698,7 +4698,7 @@ int wolfSSL_PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
}
#endif /* !NO_PWDBASED !NO_SHA*/
#if !defined(NO_PWDBASED)
#if !defined(NO_PWDBASED) && !defined(NO_HMAC)
int wolfSSL_PKCS5_PBKDF2_HMAC(const char *pass, int passlen,
const unsigned char *salt,
int saltlen, int iter,

8
wolfcrypt/src/logging.c
View File

@ -136,13 +136,6 @@ static struct log mynewt_log;
#endif /* DEBUG_WOLFSSL */
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
THREAD_LS_T int wc_svr_count = 0;
THREAD_LS_T const char *wc_svr_last_file = NULL;
THREAD_LS_T int wc_svr_last_line = -1;
#endif
/* allow this to be set to NULL, so logs can be redirected to default output */
int wolfSSL_SetLoggingCb(wolfSSL_Logging_cb f)
{
@ -1546,4 +1539,3 @@ void WOLFSSL_ERROR_MSG(const char* msg)
}
#endif /* DEBUG_WOLFSSL || WOLFSSL_NGINX || WOLFSSL_HAPROXY */

39
wolfcrypt/src/memory.c
View File

@ -31,7 +31,7 @@
#define WOLFSSL_NEED_LINUX_CURRENT
#endif
#include <wolfssl/wolfcrypt/settings.h>
#include <wolfssl/wolfcrypt/types.h>
/*
Possible memory options:
@ -1231,7 +1231,6 @@ void* wolfSSL_Realloc(void *ptr, size_t size, void* heap, int type)
/* Example for user io pool, shared build may need definitions in lib proper */
#include <wolfssl/wolfcrypt/types.h>
#include <stdlib.h>
#ifndef HAVE_THREAD_LS
@ -1439,6 +1438,42 @@ void __attribute__((no_instrument_function))
}
#endif
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
THREAD_LS_T int wc_svr_count = 0;
THREAD_LS_T const char *wc_svr_last_file = NULL;
THREAD_LS_T int wc_svr_last_line = -1;
THREAD_LS_T int wc_debug_vector_registers_retval =
WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL;
#ifdef DEBUG_VECTOR_REGISTER_ACCESS_FUZZING
WOLFSSL_LOCAL int SAVE_VECTOR_REGISTERS2_fuzzer(void) {
static THREAD_LS_T struct drand48_data wc_svr_fuzzing_state;
static THREAD_LS_T int wc_svr_fuzzing_seeded = 0;
long result;
if (wc_debug_vector_registers_retval)
return wc_debug_vector_registers_retval;
if (wc_svr_fuzzing_seeded == 0) {
long seed = WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED;
char *seed_envstr = getenv("WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED");
if (seed_envstr)
seed = strtol(seed_envstr, NULL, 0);
(void)srand48_r(seed, &wc_svr_fuzzing_state);
wc_svr_fuzzing_seeded = 1;
}
(void)lrand48_r(&wc_svr_fuzzing_state, &result);
if (result & 1)
return IO_FAILED_E;
else
return 0;
}
#endif
#endif
#ifdef WOLFSSL_LINUXKM
#include "../../linuxkm/linuxkm_memory.c"
#endif

4
wolfcrypt/src/pwdbased.c
View File

@ -559,6 +559,10 @@ int wc_PKCS12_PBKDF_ex(byte* output, const byte* passwd, int passLen,
#endif /* HAVE_PKCS12 */
#ifdef HAVE_SCRYPT
#ifdef NO_HMAC
#error scrypt requires HMAC
#endif
/* Rotate the 32-bit value a by b bits to the left.
*
* a 32-bit value.

478
wolfcrypt/test/test.c
View File

@ -574,7 +574,9 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t openssl_evpSig_test(void);
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pbkdf1_test(void);
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pkcs12_test(void);
#if defined(HAVE_PBKDF2) && !defined(NO_SHA256) && !defined(NO_HMAC)
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pbkdf2_test(void);
#endif
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t scrypt_test(void);
#ifdef HAVE_ECC
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t ecc_test(void);
@ -9466,6 +9468,19 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(c2, buf, sizeof(c2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesXtsEncrypt(aes, buf, p2, sizeof(p2), i2, sizeof(i2));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(c2, buf, sizeof(c2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
XMEMSET(buf, 0, sizeof(buf));
wc_AesXtsFree(aes);
@ -9482,6 +9497,19 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(c1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesXtsEncrypt(aes, buf, p1, sizeof(p1), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(c1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
/* partial block encryption test */
XMEMSET(cipher, 0, sizeof(cipher));
ret = wc_AesXtsEncrypt(aes, cipher, pp, sizeof(pp), i1, sizeof(i1));
@ -9492,6 +9520,21 @@ static wc_test_ret_t aes_xts_128_test(void)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(cp2, cipher, sizeof(cp2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
XMEMSET(cipher, 0, sizeof(cipher));
ret = wc_AesXtsEncrypt(aes, cipher, pp, sizeof(pp), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(cp2, cipher, sizeof(cp2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
wc_AesXtsFree(aes);
/* partial block decrypt test */
@ -9509,6 +9552,20 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(pp, buf, sizeof(pp)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, cipher, sizeof(pp), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(pp, buf, sizeof(pp)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
/* NIST decrypt test vector */
XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, c1, sizeof(c1), i1, sizeof(i1));
@ -9520,6 +9577,20 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(p1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, c1, sizeof(c1), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(p1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
/* fail case with decrypting using wrong key */
XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, c2, sizeof(c2), i2, sizeof(i2));
@ -10368,17 +10439,83 @@ static wc_test_ret_t aesecb_test(void)
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(enc, niKey, sizeof(niKey), cipher, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbEncrypt(enc, cipher, niPlain, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
ret = wc_AesEcbEncrypt(enc, cipher, niPlain, AES_BLOCK_SIZE);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
ret = wc_AesSetKey(enc, niKey, sizeof(niKey), cipher, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbEncrypt(enc, cipher, niPlain, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
XMEMSET(plain, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
}
out:
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
@ -11009,6 +11146,86 @@ static wc_test_ret_t aesctr_test(Aes* enc, Aes* dec, byte* cipher, byte* plain)
#endif
}
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
for (i = 0; i < AES_CTR_TEST_LEN; i++) {
if (testVec[i].key != NULL) {
ret = wc_AesSetKeyDirect(enc, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
/* Ctr only uses encrypt, even on key setup */
ret = wc_AesSetKeyDirect(dec, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
}
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCtrEncrypt(enc, cipher, testVec[i].plain, testVec[i].len);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCtrEncrypt(dec, plain, cipher, testVec[i].len);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
if (XMEMCMP(plain, ctrPlain, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#if !(FIPS_VERSION_EQ(2,0) && defined(WOLFSSL_ARMASM))
if (XMEMCMP(cipher, testVec[i].cipher, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#endif
}
for (i = 0; i < AES_CTR_TEST_LEN; i++) {
if (testVec[i].key != NULL) {
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKeyDirect(enc, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
/* Ctr only uses encrypt, even on key setup */
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKeyDirect(dec, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
}
ret = wc_AesCtrEncrypt(enc, cipher, testVec[i].plain, testVec[i].len);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
ret = wc_AesCtrEncrypt(dec, plain, cipher, testVec[i].len);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
if (XMEMCMP(plain, ctrPlain, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#if !(FIPS_VERSION_EQ(2,0) && defined(WOLFSSL_ARMASM))
if (XMEMCMP(cipher, testVec[i].cipher, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#endif
}
#endif /* DEBUG_VECTOR_REGISTER_ACCESS && WC_AES_C_DYNAMIC_FALLBACK */
out:
return ret;
}
@ -11262,6 +11479,57 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes_test(void)
break;
}
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
/* Iterate from one AES_BLOCK_SIZE of bigMsg through the whole
* message by AES_BLOCK_SIZE for each size of AES key. */
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
for (keySz = 16; keySz <= 32; keySz += 8) {
for (msgSz = AES_BLOCK_SIZE;
msgSz <= sizeof(bigMsg);
msgSz += AES_BLOCK_SIZE) {
XMEMSET(bigCipher, 0, sizeof(bigMsg));
XMEMSET(bigPlain, 0, sizeof(bigMsg));
ret = wc_AesSetKey(enc, bigKey, keySz, iv, AES_ENCRYPTION);
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
ret = wc_AesSetKey(dec, bigKey, keySz, iv, AES_DECRYPTION);
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
ret = wc_AesCbcEncrypt(enc, bigCipher, bigMsg, msgSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
ret = wc_AesCbcDecrypt(dec, bigPlain, bigCipher, msgSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
if (XMEMCMP(bigPlain, bigMsg, msgSz)) {
ret = WC_TEST_RET_ENC_NC;
break;
}
}
if (ret != 0)
break;
}
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
#endif /* DEBUG_VECTOR_REGISTER_ACCESS && WC_AES_C_DYNAMIC_FALLBACK */
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
XFREE(bigCipher, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
XFREE(bigPlain, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
@ -11591,6 +11859,9 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes192_test(void)
#endif
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#ifdef HAVE_AES_DECRYPT
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher));
@ -11604,9 +11875,6 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes192_test(void)
}
#endif
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT
wc_AesFree(dec);
@ -11732,6 +12000,93 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes256_test(void)
wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT
wc_AesFree(dec);
#endif
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
ret = wc_AesSetKey(enc, key, keySz, iv, AES_ENCRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
ret = wc_AesSetKey(dec, key, keySz, iv, AES_DECRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#endif
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCbcEncrypt(enc, cipher, msg, (int) sizeof(msg));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
XMEMSET(plain, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(plain, msg, (int) sizeof(plain))) {
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#endif
#ifndef HAVE_RENESAS_SYNC
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT
wc_AesFree(dec);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(enc, key, keySz, iv, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(dec, key, keySz, iv, AES_DECRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#endif
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
ret = wc_AesCbcEncrypt(enc, cipher, msg, (int) sizeof(msg));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(plain, msg, (int) sizeof(plain))) {
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#endif
#ifndef HAVE_RENESAS_SYNC
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT
wc_AesFree(dec);
#endif
#endif
out:
@ -11800,7 +12155,6 @@ static wc_test_ret_t aesgcm_default_test_helper(byte* key, int keySz, byte* iv,
/* AES-GCM encrypt and decrypt both use AES encrypt internally */
ret = wc_AesGcmEncrypt(enc, resultC, plain, plainSz, iv, ivSz,
resultT, tagSz, aad, aadSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
@ -11813,6 +12167,23 @@ static wc_test_ret_t aesgcm_default_test_helper(byte* key, int keySz, byte* iv,
if (XMEMCMP(tag, resultT, tagSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesGcmEncrypt(enc, resultC, plain, plainSz, iv, ivSz,
resultT, tagSz, aad, aadSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (cipher != NULL) {
if (XMEMCMP(cipher, resultC, cipherSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
if (XMEMCMP(tag, resultT, tagSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
#ifdef HAVE_AES_DECRYPT
ret = wc_AesGcmSetKey(dec, key, keySz);
@ -11831,6 +12202,22 @@ static wc_test_ret_t aesgcm_default_test_helper(byte* key, int keySz, byte* iv,
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesGcmDecrypt(dec, resultP, resultC, cipherSz,
iv, ivSz, resultT, tagSz, aad, aadSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (plain != NULL) {
if (XMEMCMP(plain, resultP, plainSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#endif
#endif /* HAVE_AES_DECRYPT */
ret = 0;
@ -20202,6 +20589,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
word32 bytes;
word32 idx = 0;
WC_RNG rng;
int rng_inited = 0;
wc_Sha sha;
byte hash[WC_SHA_DIGEST_SIZE];
byte signature[40];
@ -20209,6 +20597,11 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
byte* der = 0;
#endif
#define DSA_TEST_TMP_SIZE 1024
int key_inited = 0;
int derIn_inited = 0;
#ifdef WOLFSSL_KEY_GEN
int genKey_inited = 0;
#endif
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
byte *tmp = (byte *)XMALLOC(DSA_TEST_TMP_SIZE, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
DsaKey *key = (DsaKey *)XMALLOC(sizeof *key, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
@ -20216,6 +20609,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
DsaKey *derIn = (DsaKey *)XMALLOC(sizeof *derIn, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
DsaKey *genKey = (DsaKey *)XMALLOC(sizeof *genKey, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
#endif
if ((tmp == NULL) ||
(key == NULL)
#ifdef WOLFSSL_KEY_GEN
@ -20264,6 +20658,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
ret = wc_InitDsaKey(key);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
key_inited = 1;
ret = wc_DsaPrivateKeyDecode(tmp, &idx, key, bytes);
if (ret != 0)
@ -20276,6 +20671,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
#endif
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
rng_inited = 1;
ret = wc_DsaSign(hash, signature, key, &rng);
if (ret != 0)
@ -20287,8 +20683,6 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
if (answer != 1)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
wc_FreeDsaKey(key);
#ifdef WOLFSSL_KEY_GEN
{
int derSz = 0;
@ -20296,56 +20690,39 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
ret = wc_InitDsaKey(genKey);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
genKey_inited = 1;
ret = wc_MakeDsaParameters(&rng, 1024, genKey);
if (ret != 0) {
wc_FreeDsaKey(genKey);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
}
ret = wc_MakeDsaKey(&rng, genKey);
if (ret != 0) {
wc_FreeDsaKey(genKey);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
}
der = (byte*)XMALLOC(FOURK_BUF, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
if (der == NULL) {
wc_FreeDsaKey(genKey);
if (der == NULL)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
derSz = wc_DsaKeyToDer(genKey, der, FOURK_BUF);
if (derSz < 0) {
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
if (derSz < 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(derSz), out);
}
ret = SaveDerAndPem(der, derSz, keyDerFile, keyPemFile,
DSA_PRIVATEKEY_TYPE);
if (ret != 0) {
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
wc_FreeDsaKey(genKey);
if (ret != 0)
goto out;
}
ret = wc_InitDsaKey(derIn);
if (ret != 0) {
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
wc_FreeDsaKey(genKey);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
}
derIn_inited = 1;
idx = 0;
ret = wc_DsaPrivateKeyDecode(der, &idx, derIn, derSz);
if (ret != 0) {
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
wc_FreeDsaKey(derIn);
wc_FreeDsaKey(genKey);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
}
}
#endif /* WOLFSSL_KEY_GEN */
out:
@ -20368,15 +20745,20 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
if (tmp)
XFREE(tmp, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
if (key)
if (key) {
if (key_inited)
wc_FreeDsaKey(key);
XFREE(key, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
}
#ifdef WOLFSSL_KEY_GEN
if (derIn) {
wc_FreeDsaKey(derIn);
if (derIn_inited)
wc_FreeDsaKey(derIn);
XFREE(derIn, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
}
if (genKey) {
wc_FreeDsaKey(genKey);
if (genKey_inited)
wc_FreeDsaKey(genKey);
XFREE(genKey, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
}
#endif
@ -20384,13 +20766,18 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
#else /* !WOLFSSL_SMALL_STACK || WOLFSSL_NO_MALLOC */
#ifdef WOLFSSL_KEY_GEN
wc_FreeDsaKey(derIn);
wc_FreeDsaKey(genKey);
if (key_inited)
wc_FreeDsaKey(key);
if (derIn_inited)
wc_FreeDsaKey(derIn);
if (genKey_inited)
wc_FreeDsaKey(genKey);
#endif
#endif
wc_FreeRng(&rng);
if (rng_inited)
wc_FreeRng(&rng);
return ret;
}
@ -41352,7 +41739,7 @@ static wc_test_ret_t pkcs7authenveloped_run_vectors(byte* rsaCert, word32 rsaCer
#endif
#if !defined(NO_AES) && defined(WOLFSSL_AES_256) && defined(HAVE_ECC) && \
defined(WOLFSSL_SHA512)
defined(WOLFSSL_SHA512) && defined(HAVE_AESGCM)
WOLFSSL_SMALL_STACK_STATIC const byte optionalUkm[] = {
0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07
};
@ -41455,7 +41842,12 @@ static wc_test_ret_t pkcs7authenveloped_run_vectors(byte* rsaCert, word32 rsaCer
NULL, 0, 0, NULL, 0, NULL, 0, 0, 0, 0, 0, 0, 0, 0,
"pkcs7authEnvelopedDataAES256GCM_IANDS.der");
#endif
#endif /* NO_AES */
#else /* NO_AES || !HAVE_AESGCM */
(void)rsaCert;
(void)rsaCertSz;
(void)rsaPrivKey;
(void)rsaPrivKeySz;
#endif /* NO_AES || !HAVE_AESGCM */
#endif
/* key agreement key encryption technique*/
@ -48972,7 +49364,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t cryptocb_test(void)
#endif
#endif
#ifndef NO_PWDBASED
#if defined(HAVE_PBKDF2) && !defined(NO_SHA256)
#if defined(HAVE_PBKDF2) && !defined(NO_SHA256) && !defined(NO_HMAC)
if (ret == 0)
ret = pbkdf2_test();
#endif

4
wolfssl/wolfcrypt/aes.h
View File

@ -243,13 +243,15 @@ enum {
#endif
struct Aes {
/* AESNI needs key first, rounds 2nd, not sure why yet */
ALIGN16 word32 key[60];
#ifdef WC_AES_BITSLICED
/* Extra key schedule space required for bit-slicing technique. */
ALIGN16 bs_word bs_key[15 * AES_BLOCK_SIZE * BS_WORD_SIZE];
#endif
word32 rounds;
#ifdef WC_AES_C_DYNAMIC_FALLBACK
word32 key_C_fallback[60];
#endif
int keylen;
ALIGN16 word32 reg[AES_BLOCK_SIZE / sizeof(word32)]; /* for CBC mode */

2
wolfssl/wolfcrypt/cryptocb.h
View File

@ -89,7 +89,6 @@ typedef struct wc_CryptoInfo {
#if HAVE_ANONYMOUS_INLINE_AGGREGATES
union {
#endif
#if !defined(NO_RSA) || defined(HAVE_ECC)
struct {
int type; /* enum wc_PkType */
#if HAVE_ANONYMOUS_INLINE_AGGREGATES
@ -206,7 +205,6 @@ typedef struct wc_CryptoInfo {
};
#endif
} pk;
#endif /* !NO_RSA || HAVE_ECC */
#if !defined(NO_AES) || !defined(NO_DES3)
struct {
int type; /* enum wc_CipherType */

14
wolfssl/wolfcrypt/error-crypt.h
View File

@ -238,19 +238,19 @@ enum {
BAD_LENGTH_E = -279, /* Value of length parameter is invalid. */
ECDSA_KAT_FIPS_E = -280, /* ECDSA KAT failure */
RSA_PAT_FIPS_E = -281, /* RSA Pairwise failure */
KDF_TLS12_KAT_FIPS_E = -282, /* TLS12 KDF KAT failure */
KDF_TLS13_KAT_FIPS_E = -283, /* TLS13 KDF KAT failure */
KDF_TLS12_KAT_FIPS_E = -282, /* TLS12 KDF KAT failure */
KDF_TLS13_KAT_FIPS_E = -283, /* TLS13 KDF KAT failure */
KDF_SSH_KAT_FIPS_E = -284, /* SSH KDF KAT failure */
DHE_PCT_E = -285, /* DHE Pairwise Consistency Test failure */
ECC_PCT_E = -286, /* ECDHE Pairwise Consistency Test failure */
FIPS_PRIVATE_KEY_LOCKED_E = -287, /* Cannot export private key. */
PROTOCOLCB_UNAVAILABLE = -288, /* Protocol callback unavailable */
AES_SIV_AUTH_E = -289, /* AES-SIV authentication failed */
NO_VALID_DEVID = -290, /* no valid device ID */
AES_SIV_AUTH_E = -289, /* AES-SIV authentication failed */
NO_VALID_DEVID = -290, /* no valid device ID */
IO_FAILED_E = -291, /* Input/output failure */
SYSLIB_FAILED_E = -292, /* System/library call failed */
USE_HW_PSK = -293, /* Callback return to indicate HW has PSK */
IO_FAILED_E = -291, /* Input/output failure */
SYSLIB_FAILED_E = -292, /* System/library call failed */
USE_HW_PSK = -293, /* Callback return to indicate HW has PSK */
ENTROPY_RT_E = -294, /* Entropy Repetition Test failed */
ENTROPY_APT_E = -295, /* Entropy Adaptive Proportion Test failed */

166
wolfssl/wolfcrypt/memory.h
View File

@ -251,9 +251,173 @@ WOLFSSL_LOCAL void wc_MemZero_Add(const char* name, const void* addr,
WOLFSSL_LOCAL void wc_MemZero_Check(void* addr, size_t len);
#endif
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
WOLFSSL_API extern THREAD_LS_T int wc_svr_count;
WOLFSSL_API extern THREAD_LS_T const char *wc_svr_last_file;
WOLFSSL_API extern THREAD_LS_T int wc_svr_last_line;
#ifdef DEBUG_VECTOR_REGISTERS_ABORT_ON_FAIL
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE abort();
#elif defined(DEBUG_VECTOR_REGISTERS_EXIT_ON_FAIL)
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE exit(1);
#else
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE
#endif
#define SAVE_VECTOR_REGISTERS(fail_clause) { \
int _svr_ret = wc_debug_vector_registers_retval; \
if (_svr_ret != 0) { fail_clause } \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
}
WOLFSSL_API extern THREAD_LS_T int wc_debug_vector_registers_retval;
#ifndef WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL
#define WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL 0
#endif
#define WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(x) do { \
if (((x) != 0) && (wc_svr_count > 0)) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_debug_vector_registers_retval = (x); \
} while (0)
#ifdef DEBUG_VECTOR_REGISTER_ACCESS_FUZZING
#ifndef WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED
#define WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED 0
#endif
WOLFSSL_LOCAL int SAVE_VECTOR_REGISTERS2_fuzzer(void);
#define SAVE_VECTOR_REGISTERS2(...) ({ \
int _svr2_val = SAVE_VECTOR_REGISTERS2_fuzzer(); \
if (_svr2_val == 0) { \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
_svr2_val = 0; \
} \
_svr2_val; \
})
#else
#define SAVE_VECTOR_REGISTERS2(...) ({ \
int _svr2_val; \
if (wc_debug_vector_registers_retval != 0) { \
if (wc_svr_count > 0) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
_svr2_val = wc_debug_vector_registers_retval; \
} else { \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
_svr2_val = 0; \
} \
_svr2_val; \
})
#endif
#define ASSERT_SAVED_VECTOR_REGISTERS(fail_clause) do { \
if (wc_svr_count <= 0) { \
fprintf(stderr, \
("ASSERT_SAVED_VECTOR_REGISTERS : %s @ L%d : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
} while (0)
#define ASSERT_RESTORED_VECTOR_REGISTERS(fail_clause) do { \
if (wc_svr_count != 0) { \
fprintf(stderr, \
("ASSERT_RESTORED_VECTOR_REGISTERS : %s @ L%d" \
" : wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
} while (0)
#define RESTORE_VECTOR_REGISTERS(...) do { \
--wc_svr_count; \
if ((wc_svr_count > 4) || (wc_svr_count < 0)) { \
fprintf(stderr, \
("%s @ L%d : decr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
} while(0)
#endif
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* WOLFSSL_MEMORY_H */

100
wolfssl/wolfcrypt/types.h
View File

@ -1566,90 +1566,24 @@ typedef struct w64wrapper {
#define PRAGMA_DIAG_POP /* null expansion */
#endif
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
WOLFSSL_API extern THREAD_LS_T int wc_svr_count;
WOLFSSL_API extern THREAD_LS_T const char *wc_svr_last_file;
WOLFSSL_API extern THREAD_LS_T int wc_svr_last_line;
#ifdef DEBUG_VECTOR_REGISTERS_ABORT_ON_FAIL
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE abort();
#elif defined(DEBUG_VECTOR_REGISTERS_EXIT_ON_FAIL)
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE exit(1);
#else
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE
#endif
#define SAVE_VECTOR_REGISTERS(...) { \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
"%s @ L%d : incr : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
}
#define ASSERT_SAVED_VECTOR_REGISTERS(fail_clause) { \
if (wc_svr_count <= 0) { \
fprintf(stderr, \
"ASSERT_SAVED_VECTOR_REGISTERS : %s @ L%d : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
}
#define ASSERT_RESTORED_VECTOR_REGISTERS(fail_clause) { \
if (wc_svr_count != 0) { \
fprintf(stderr, \
"ASSERT_RESTORED_VECTOR_REGISTERS : %s @ L%d : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
}
#define RESTORE_VECTOR_REGISTERS(...) { \
--wc_svr_count; \
if ((wc_svr_count > 4) || (wc_svr_count < 0)) { \
fprintf(stderr, \
"%s @ L%d : decr : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
}
#else
#ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef ASSERT_SAVED_VECTOR_REGISTERS
#define ASSERT_SAVED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef ASSERT_RESTORED_VECTOR_REGISTERS
#define ASSERT_RESTORED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() WC_DO_NOTHING
#endif
#ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef SAVE_VECTOR_REGISTERS2
#define SAVE_VECTOR_REGISTERS2() 0
#endif
#ifndef WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL
#define WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(x) WC_DO_NOTHING
#endif
#ifndef ASSERT_SAVED_VECTOR_REGISTERS
#define ASSERT_SAVED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef ASSERT_RESTORED_VECTOR_REGISTERS
#define ASSERT_RESTORED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() WC_DO_NOTHING
#endif
#if FIPS_VERSION_GE(5,1)
#define WC_SPKRE_F(x,y) wolfCrypt_SetPrivateKeyReadEnable_fips((x),(y))