matti

I’m getting like 10-20tps with some other similar-sized models, but didn’t benchmark qwen yet. Note that they are mixture-of-experts, so they are evaluating only a few billion parameters each time.

Edit:

I benchmarked the model now:

prompt eval count:    11 token(s)  
prompt eval duration: 485.605ms  
prompt eval rate:     22.65 tokens/s  
eval count:           207 token(s)  
eval duration:        21.392268s  
eval rate:            9.68 tokens/s  

Not bad for a good CPU model.

Please ignore the post if you are not interested.