bug: SFTTrainer experiments that use fms_hf_tuning_version==3.0.0 do not work with vision models

[VassilisVassiliadis]

Issue Description

SFTTrainer experiments that use fms_hf_tuning_version==3.0.0 do not work with vision models.

How to reproduce

Create a space with a SFTrainer experiment that uses fms_hf_tuning_version==3.0.0 e.g.

sampleStoreIdentifier: "8be994"

experiments:
- experimentIdentifier: finetune_full_benchmark-v1.0.0
  actuatorIdentifier: SFTTrainer
  parameterization:
    - property:
        identifier: fms_hf_tuning_version
      value: "3.0.0"
    - property:
        identifier: dataset_image_field
      value: images
    - property:
        identifier: remove_unused_columns
      value: false
    - property:
        identifier: dataset_kwargs_skip_prepare_dataset
      value: true
    - property:
        identifier: accelerate_config_fsdp_transformer_layer_cls_to_wrap
      value: MistralDecoderLayer
    - property:
        identifier: num_train_epochs
      value: 10
    - property:
        identifier: auto_stop_method
      value: WARMUP_60S_STABLE_120S_OR_10_STEPS
    - property:
        identifier: dataset_id
      value: vision-384x384-16384plus-entries-4096

entitySpace:
  - identifier: "model_name"
    propertyDomain:
      values: [ 'llava-v1.6-mistral-7b' ]
  - identifier: "number_gpus"
    propertyDomain:
      values: [ 1, 2, 4, 8 ]
  - identifier: "model_max_length"
    propertyDomain:
      values: [ 512, 1024, 2048, 4096, 8192]
  - identifier: "batch_size"
    propertyDomain:
      values: [ 1, 2, 4, 8, 16, 32, 128, 256, 512 ]
  - identifier: "gpu_model"
    propertyDomain:
      values: [ 'NVIDIA-A100-SXM4-80GB' ]

Start the operation and then you'll see the measurement fail with the exception:

TypeError: apply_tokenizer_chat_template() missing 1 required positional argument: 'formatted_text_column_name'

Expected behaviour

I should be able to run vision models with fms_hf_tuning_version==3.0.0 successfully or get an invalid measurement because of a known incompatibility of fms-hf-tuning with vision models.

Additional information

This is due to a bug introduced in fms-hf-tuning==3.0.0 - details: foundation-model-stack/fms-hf-tuning#606

[VassilisVassiliadis]

We're going to open a small PR that will add a line in the docs of the sfttrainer actuator that specifies vision models are not working reliably with fms-hf-tuning.

[VassilisVassiliadis]

The fms-hf-tuning devs are working on a fix: foundation-model-stack/fms-hf-tuning#671

We'll wait for the next release of fms-hf-tuning and try it out before we update our docs.

[AlessandroPomponio]

The PR linked above has been merged, but there hasn't been a release containing the fix, which was merged 2 months ago. Should we ping the fms-hf-tuning maintainers?

[AlessandroPomponio]

Should we follow up on this or close it?

[VassilisVassiliadis]

We don't currently have a need for vision models. Let's keep this issue as the record that they're currently not supported but close it without making any changes.