Build a benchmark for sotopia!

[XuhuiZhou]

We are currently working on building a benchmark for Sotopia!

@clementou is currently in charge of adapting the Sotopia codebase to accommodate as many models as possible!
We are looking for help in designing the UI for the Sotopia benchmark.

Some inspirations for the UI:

• https://huggingface.co/spaces/allenai/WildBench
• https://huggingface.co/spaces/lmsys/mt-bench

[Algoroxyolo]

what models are we looking at?

I am thinking about a few: qwen series, Mixtral8x7b, llama2 series

[clementou]

Switched to using the LiteLLM Library so we can use any API endpoints supported there.

Having some issues right now calculating model performance since writing episodes still uses the model_dict to write the model name. So the model name written to the episode doesn't match the model being used. Can't pinpoint where this happens right now, but a test case for this would help.

Haven't started on the UI yet but should be fairly easy.

[XuhuiZhou]

Thanks @clementou! Maybe @Algoroxyolo you could start working on the UI?

[Algoroxyolo]

I am not so experienced with UI but I can try

[Algoroxyolo]

Where should I start looking at?

[XuhuiZhou]

Here https://huggingface.co/spaces/allenai/WildBench!

[XuhuiZhou]

Also- do you mind if I get a few more people of interest involved?

@Algoroxyolo
Yeah! As long as they are happy to contribute :)