Comparing models on Turkish reviews: API and local model
We compared a hosted API (Jev), a pretrained local model (Laya) and the same model after full fine-tuning (Laya-FT) on binary Turkish sentiment, with the same fixed inputs: 15,000 test reviews and 40,000 training reviews.
Business question
For a narrowly defined classification task, how much does task-specific training improve a locally deployed model, and does that improvement hold beyond the collection it was trained on?
Method
All three systems received the same Turkish question and the same fixed inputs under one frozen protocol. The local model was fully fine-tuned on 40,000 reviews; the checkpoint was chosen on a separate 5,000-review development set using a predeclared rule. Test labels were used only for scoring. Macro-F1 is computed against source labels; 95% intervals come from 2,000 stratified bootstrap replicates.
Key findings
- Fine-tuning raised the local model's macro-F1 on the main test (10,000 reviews) from 77.74% to 93.26%, a gain of 15.52 points.
- On the held-out collection (5,000 reviews) the gain was smaller, from 76.29% to 83.72%, or 7.43 points.
- Jev scored higher on both partitions (93.59% and 88.86%). The 0.32-point main-test gap is not evidence of a definite advantage; on the held-out collection the gap is 5.14 points.
- The aggregate score alone is not enough. On the held-out collection, the share of negative reviews caught fell from 92.60% to 75.36% after fine-tuning.
- All three systems produced all 15,000 predictions without technical failures or missing responses. The calculated cost of the Jev API usage was about USD 0.28.
Limitations
- Source labels (star ratings) have not been independently validated by humans; a star rating can disagree with the text.
- One training seed, two collections and one binary question; no paired significance test was run for the small main-test gap.
- The platform independence of the held-out collection is not verified, and pretraining contamination cannot be ruled out.
- Latency figures describe these runs; they are not a controlled speed comparison.
- Not peer reviewed; the study does not validate production performance for any project.
Results
Macro-F1 against source labels (percent; higher is better).
| Model | Main test (10,000) | Held-out collection (5,000) |
|---|---|---|
| Jev API | 93.59 | 88.86 |
| Laya (pretrained) | 77.74 | 76.29 |
| Laya-FT (full fine-tuning) | 93.26 | 83.72 |
Detailed classification results; P and R are negative-class precision and recall.
| Model | Test | Accuracy | Macro-F1 | 95% interval | P negative | R negative |
|---|---|---|---|---|---|---|
| Jev | Main | 93.59 | 93.59 | 93.06–94.07 | 95.70 | 91.28 |
| Laya | Main | 78.41 | 77.74 | 76.96–78.54 | 71.10 | 95.72 |
| Laya-FT | Main | 93.27 | 93.26 | 92.78–93.75 | 96.02 | 90.28 |
| Jev | Held-out | 88.90 | 88.86 | 88.00–89.67 | 93.83 | 83.28 |
| Laya | Held-out | 76.88 | 76.29 | 75.11–77.44 | 70.45 | 92.60 |
| Laya-FT | Held-out | 83.84 | 83.72 | 82.74–84.77 | 90.75 | 75.36 |
Intervals describe test-sample uncertainty, not variation across training seeds.
Fine-tuning changed the error balance
On the held-out collection, Laya-FT became far more precise in its negative predictions (precision 70.45% → 90.75%) but caught fewer negative reviews (recall 92.60% → 75.36%). A higher aggregate score therefore does not mean fewer missed negative reviews. Where missing negative feedback is costly, the model should not be chosen on macro-F1 alone.
Data and partitions
The main collection is MüşteriYorumları, a set of star-rated e-commerce reviews: one or two stars were mapped to negative, four or five to positive, and three-star reviews were excluded. The second test collection is Turkish Product Reviews, which was used in none of the training, development or selection steps. Cleaning removed empty and malformed records, exact duplicates, conflicting labels and groups shared across collections; near duplicates were searched with MinHash/LSH. Email addresses, phone numbers and URLs were masked.
| Partition | Collection | Negative | Positive | Total |
|---|---|---|---|---|
| Training | Main | 14,253 | 25,747 | 40,000 |
| Development | Main | 2,500 | 2,500 | 5,000 |
| Calibration (unused) | Main | 2,500 | 2,500 | 5,000 |
| Main test | Main | 5,000 | 5,000 | 10,000 |
| Held-out test | Second | 2,500 | 2,500 | 5,000 |
Only the training set is class-imbalanced; no synthetic examples or class weighting were used.
Training and measurement
Laya-FT was produced by training all parameters of Laya Multilingual (about 322 million) on a single NVIDIA A100 GPU. After four epochs, the checkpoint with the best development macro-F1 (93.66%) was selected. Test results played no part in any selection.
The local models ran on the same Apple M1 Pro laptop. Median single-review latency was about 50–56 ms for the local models and about 290–755 ms for the Jev API, including network overhead. Because conditions differ, these figures should not be read as a speed advantage.
Conclusion
Under a frozen 15,000-review protocol, full fine-tuning moved the local model much closer to the API on the main collection and also improved its held-out score; Jev kept the higher observed score on both partitions. The defensible conclusion is a substantial, task-specific improvement with remaining sensitivity to collection differences, not general parity or a controlled speed advantage.
Next steps: additional baselines, repeated training seeds and a blind, human-labelled subset.
Sources and reproducibility
- Jev 1.13.0 — TypeSafe AI model documentation
- Laya Multilingual — Convai Innovations model card
- MüşteriYorumları — Turkish NLP Suite (CC-BY-SA-4.0)
- Turkish Product Reviews — F. Barmanbay
- Laya source code and fine-tuning implementation, version 0.3.20
No raw review text or model weights are shared. Partitions, source revisions, model hashes and example-level predictions are kept in the local study archive. AI assistance was used for implementation, analysis and drafting; generated labels were not used as ground truth. The full technical note is available as a PDF.