So, we are currently running into issues with extremly slow loading times. We have data that takes minutes to load, even on relatively strong hardware with NVMe SSDs.
In order to reduce the amount of stored data, I investigated how certain data is stored.
I noticed that String is still being saved as char[]. A char uses 16-bit in Java, instead of 8-bit typically required per character for Latin languages using UTF-8 encoding.
This means, that storage usage with databases using predominantly strings can be slashed by 40-50%.
I am assuming that it is quite hard to change this now, as it'd require a legacy type handler for strings? Hence, I understand if you don't want to do this on your side.
If you do NOT want to change this, maybe you can answer these questions:"
- Would this require both a
LegacyTypeHandler (for conversion) and a TypeHandler for new and already converted String elements
- Would this impose a permanent runtime speed penality, as data is always converted lazily as far as I understand?
Also, if you think this would be a potentially sensible thing to do AND don't want to do this, maybe this should be mentioned as potential performance tuning in the documentation, as I think others could potentially greatly benefit from this as well.
So, we are currently running into issues with extremly slow loading times. We have data that takes minutes to load, even on relatively strong hardware with NVMe SSDs.
In order to reduce the amount of stored data, I investigated how certain data is stored.
I noticed that
Stringis still being saved aschar[]. Acharuses 16-bit in Java, instead of 8-bit typically required per character for Latin languages using UTF-8 encoding.This means, that storage usage with databases using predominantly strings can be slashed by 40-50%.
I am assuming that it is quite hard to change this now, as it'd require a legacy type handler for strings? Hence, I understand if you don't want to do this on your side.
If you do NOT want to change this, maybe you can answer these questions:"
LegacyTypeHandler(for conversion) and aTypeHandlerfor new and already convertedStringelementsAlso, if you think this would be a potentially sensible thing to do AND don't want to do this, maybe this should be mentioned as potential performance tuning in the documentation, as I think others could potentially greatly benefit from this as well.