CNDLanguageModelAccuracyTask
Language Model Accuracy
Team
Dev: Vladimir Voskresensky (VV; NB nick: vv159170), Nick Krasilnikov (NK; NB nick: nnnnnk)
QA: Alexander Ioffe (NB nick: aioffe), Dmitry Nikitin (NB nick: dnikitin)
Lead: Vladimir Voskresensky
Charter
Improve accuracy of Language Model and services based on it.
User View
not acceptable
Metrics
Metrics for language model accuracy
Goals
The goal is to have 95% accuracy of Language Model.
Test Plan
Schedule
Dev
| Milestone | Date | Status | Content |
|---|---|---|---|
| M1 | Late May, Early June | DONE | Provide data for accuracy measurement |
| M2 | 15 June | IN PROGRESSS | Analyze numbers and failures to detect problem areas |
| M3 | from June till ... | IN PROGRESS | Fix top accuracy IZs. Analyze new top IZs and fix them |
QA
| Milestone | Date | Status | Content |
|---|---|---|---|
| M0.1 | June, 2 | DONE | Prepare Perl project for testing |
| M0.2 | June, 4 | DONE | Prepare Python project for testing |
| M0.3 | June, 6 | DONE | Prepare Povray project for testing |
| M1 | June, 6 | DONE | Prepare infrastructure to determine accuracy number of the test project. |
| M2 | June, 15 | IN PROGRESS | Provide Code Completion Numbers for selected projects => do that daily |
| M2 | June, 15 | Provide Find Usages numbers for selected projects => do that daily |
Risks and dependencies
Depend on accuracy measurement results. Results of test runs must be stable and without fail alarms. Tests run slowly and to provide results daily needs number of configured machines (in progress)
Status and test results
Projects for accuracy measurement
We are using different projects to cover real usage of C/C++
- C++ Projects
| Not intensive macro usages | Intensive macro usages | |
|---|---|---|
| Not intensive template usages | litesql (98.8%, templ 0.2%) (8900 CP), Quote (100%)(700 CP) | Freeway (mixed a little) (99.5) (4200 CP), Povray (99.8%, templ 0.1%) (13 000 CP) |
| Intensive template usages | loki (98%, templ 1.5%) (a little macros) (7200 CP) | CLucene (99.2%, templ 1.5%) (7 000 CP), boost (96.6%, templ 1.7%) (163000 CP) |
- C Projects
| Not intensive macro usages | Intensive macro usages |
|---|---|
| TBD | Perl (97.5%) (10500 CP), Python (97%) (12 000 CP ), OS.UTS (97.2%) |
- Mixed project (C/C++)
- MySQL (99.3%, templ 0.2%) (650 000 CP) - lot of templates and lot of macros
- ddd (99%) (*170 000 CP) - few templates and lot of macros
Design secifications
Unit tests
Issuezilla
Feature IZ link
Status Whiteboard keyword: model_accuracy
(click on the keyword above to get issues list)
