As for models, we support evaluating a wide range of models including Llama - in addition we support OpenAI, Anthropic, Gemini, Mistral, MosaicML, Together, and self-hosted models. We are also compatible with frameworks such as Langchain, Magentic and LiteLLM.
For how to integrate different models please see here:
https://github.com/log10-io/log10
Also wanted to address the confusion on the role of llama-3-8b and llama-3.1-8b. In the blog post we use these models as an example of an evaluator LLM. We select what is best for your custom eval under the hood. LSR is just one example of the research powering our custom evals
The LSR version of AutoFeedback is on our Pro tier. You can sign up for more info on that here: https://tally.so/r/w2XVQA
In the free trial of Log10.io you can use the ICL (self-improving) version of AutoFeedback for hallucination detection and RAG evals. In generalize, you can customize evals for a wide range of use cases using AutoFeedback: https://docs.log10.io/feedback/auto_feedback https://github.com/log10-io/log10-cookbook/tree/main https://arjunbansal.substack.com/p/hybrid-evaluation-scaling...
As for models, we support evaluating a wide range of models including Llama - in addition we support OpenAI, Anthropic, Gemini, Mistral, MosaicML, Together, and self-hosted models. We are also compatible with frameworks such as Langchain, Magentic and LiteLLM. For how to integrate different models please see here: https://github.com/log10-io/log10