Just never hit the performance goals..

[neocoretechs]

I used the Oracle Zurich version of Llama pure Java inference with vector extensions and then played with trying to boost inference. For some small Llama models it was on a par with native Llama.cpp but never quite reached the same inference TPS numbers otherwise. Hope you have better luck!
https://github.com/neocoretechs/Llama4jNoDeps

[bsbodden]

Thanks @neocoretechs - still just experimenting and refining... I'll check your repo to see if there are lessons to be learned. I am mostly want to tackle small models, and small finetuned models for specific verticals (medical, banking, insurance, fintech, etc.)

[bsbodden]

@neocoretechs what I did was to push the Java implementation as far as I could, with several benchmarks to measure different aspect of the candidate models. The release gate is based on observed correctness and user-facing latency. Java 25 Vector API and HotSpot code generation cannot express or consistently lower every quantized inference kernel close enough to mature native runtimes.
For below acceptable threshold model profiles, I added a backend-native JAR that adds the missing throughput with a small Rust binary. I'm seeing this as a temporary performance compatibility layer, a shim we can drop off later when the JVM catches up in that particular aspect.

[bsbodden]

@neocoretechs would you mind giving it a try... https://modeljars.org

[neocoretechs]

@neocoretechs would you mind giving it a try... https://modeljars.org

Sure, I'll try some models, but I looked at the code and didnt see where I could access the inference pipeline. For my use case I need access to the context window, the chat template, tokenizer, etc. Thats why I built the FFI layer into llama.cpp. Maybe thats out of scope for the project, but if the idea is to provide an alternative to llama.cpp the API is critical. Thanks.