[BUG] OQS_KEM_new/OQS_SIG_new NULL return stored as handle — deferred crash on invalid algorithm

[Fuyugithub]

Bug Description

In the liboqs JNI bindings, OQS_KEM_new and OQS_SIG_new can return NULL when the algorithm name is invalid or unsupported. The NULL pointer is stored directly into the Java object's native handle field via setHandle. Every subsequent JNI call that retrieves this handle and dereferences it will crash with a NULL pointer dereference.

Location

liboqs-android/jni/jni/KeyEncapsulation.c, lines 14-18
liboqs-android/jni/jni/Signature.c, lines 14-18

Code

// KeyEncapsulation.c:
JNIEXPORT void JNICALL Java_net_ivpn_liboqs_KeyEncapsulation_create_1KEM_1new
  (JNIEnv *env, jobject obj, jstring jstr)
{
    const char *str_native = (*env)->GetStringUTFChars(env, jstr, 0);
    OQS_KEM *kem = OQS_KEM_new(str_native);           // can return NULL
    (*env)->ReleaseStringUTFChars(env, jstr, str_native);
    setHandle(env, obj, kem, "native_kem_handle_");    // stores NULL
}

Later, any operation retrieves the handle and dereferences it:

// generate_keypair:
OQS_KEM *kem = (OQS_KEM *) getHandle(env, obj, "native_kem_handle_");
OQS_STATUS rv_ = OQS_KEM_keypair(kem, ...);   // NULL dereference → SIGSEGV

Same pattern in Signature.c with OQS_SIG_new.

Impact

Passing an unsupported algorithm name (e.g., a typo or an algorithm disabled at compile time) silently stores a NULL handle. The crash happens later — on key generation, encapsulation, signing, or any other operation — far from the actual failure point. This makes the bug very hard to diagnose and is trivially triggerable from Java.

Suggested Fix

OQS_KEM *kem = OQS_KEM_new(str_native);
(*env)->ReleaseStringUTFChars(env, jstr, str_native);
if (kem == NULL) {
    jclass exc = (*env)->FindClass(env, "java/lang/IllegalArgumentException");
    (*env)->ThrowNew(env, exc, "Unsupported or disabled KEM algorithm");
    return;
}
setHandle(env, obj, kem, "native_kem_handle_");

Environment

• IVPN Android: current main branch
• Files: liboqs-android JNI bindings (KeyEncapsulation.c, Signature.c)
• Affects: all post-quantum KEM and signature operations