- [Pending Merge] LangChain4j integration
- Additional quantization formats
- Q8
- Q4
- INT8 native support for GPUs
- Additional architectures and model format
- Mistral/Mixtral models
- Qwen
- Phi-3
- Gemma/Gemma2 models
- TinyLlamas
- SafeTensors format
- PyTorch checkpoint loading
- Automatic model conversion utilities
- Advanced inference capabilities
- Batch inference support
- Speculative decoding
- Performance optimizations
- Multi-GPU support
- Memory-efficient attention mechanisms
- More Kernel fusion improvements