Performance benchmarking of free LLMs: A practical analysis of token processing efficiency

Abstract

Background: With the rapid advancement of large language models (LLMs), comparative performance evaluations have become essential to guide both research and real-world deployment. Differences in processing efficiency, cost, and scalability significantly influence their practical applications.

Purpose: The study aims to investigate and compare the performance of four leading LLMs—Llama 3.3 8B Instruct (Meta), Gemma 3n 4B (Google AI Studio), DeepSeek Prover V2, and DeepHermes 3 Llama 3 8B Preview (Chutes)—across selected computational metrics.

Research Design: The research employed a comparative experimental approach, where models were tested under varying token loads. Key evaluation parameters included token count, processing speed (tokens per second), execution duration, and cost efficiency.

Study Sample: The sample comprised multiple performance instances for each of the four selected models, tested under controlled computational conditions to ensure consistency.

Data Collection and/or Analysis: Performance statistics were collected systematically for each model. Descriptive analysis was carried out by comparing throughput rate, average response times, and relative cost differences.

Results: Findings revealed substantial variation across the models. Llama 3.3 and DeepHermes consistently recorded the fastest throughput, exceeding 190 tokens per second, with response times under 4 seconds. Gemma 3n showed slower throughput at higher token counts, reflecting a trade-off between speed and token management. DeepSeek Prover V2 performed moderately across all metrics but lagged behind in speed compared to Llama and DeepHermes. The results suggest that model selection should be context-specific, with Llama 3.3 and DeepHermes offering the best trade-off between speed and cost-effectiveness for practical NLP tasks, while other models may be more suited for research-oriented applications.