Then it wouldn't be a lexical closure.

It can be from the perspective of the calling code.

Roughly speaking:

  thread_local! {
    static CBQ: Option<Box<impl FnMut(i32, i32) -> i32>>;
  }

  #[no_mangle]
  extern "C" fn qsort(array: *mut i32, val: usize, callback: impl FnMut(i32, i32) -> i32);

  pub fn rust_qsort(array : Vec<i32>, callback: impl FnMut(i32, i32) -> i32){
    CBQ.replace(Box::new(callback)).unwrap_none();
    
    unsafe {
        qsort(array.as_mut_ptr(), array.len(), &rust_qsort_callback);
    }
    
    CBQ.take().unwrap();
  }

  fn rust_qsort_callback(a: *mut i32, b: *mut i32) -> i32 {
    let callback = CBQ.take().unwrap();

    let (a, b) = unsafe { 
        (*a, *b)
    };

    let result = callback(a, b);
    
    CBQ.replace(callback).unwrap_none();

    result
  }

  fn main() {
    let a = vec![4,5,6,3,2];

    rust_qsort(a, |a, b| {
        if a < b {
            -1
        } else if a > b {
            1
        } else {
            0
        }
    })
  }

ought to work. (There's some fun with generics and panics, which is some fun to solve, but nothing which breaks the premise above).

This fails horribly if called recursively (or from a signal handler). You need something like:

  wrapped_qsort(/* array,callback */)
    {
    auto tmp = CBQ;
    CBQ = wrap(callback);
    qsort(array.ptr,array.len,cbq_callback);
    CBQ = tmp; /* pop old value from stack */
    }

It'll fail from a signal handler inside qsort, or inside the callback function.

It won't fail recursively - while the second call is happening, the first will be stored on the stack (see the take and replace in the callback shim function)

Your `fn rust_qsort` takes ownership of the vector, so it frees its memory after sorting and it can't be used after sorting in the caller function. And generic `impl FnMut` won't work in `extern "C"`, it only accepts function pointers.

you have reinvented dynamic scoping :).