CNDLanguageModelAccuracyTask

Language Model Accuracy

Team

Dev: Vladimir Voskresensky (VV; NB nick: vv159170), Nikolay Krasilnikov (NK; NB nick: nnnnnk)

QA: Alexander Ioffe (NB nick: aioffe), Dmitry Nikitin (NB nick: dnikitin)

Lead: Vladimir Voskresensky

Charter

Improve accuracy of Language Model and services based on it.

User View

not acceptable

Metrics

Metrics for language model accuracy

Goals

The goal is to have 95% accuracy of Language Model.

Test Plan

Plan

Schedule

Dev

Milestone Date Status Content
M1 Late May, Early June DONE Provide data for accuracy measurement
M2 15 June IN PROGRESSS Analyze numbers and failures to detect problem areas
M3 from June till ... IN PROGRESS Fix top accuracy IZs. Analyze new top IZs and fix them


QA

Milestone Date Status Content
M0.1 June, 2 DONE Prepare Perl project for testing
M0.2 June, 4 DONE Prepare Python project for testing
M0.3 June, 6 DONE Prepare Povray project for testing
M1 June, 6 DONE Prepare infrastructure to determine accuracy number of the test project.
M2 June, 15 IN PROGRESS Provide Code Completion Numbers for selected projects => do that daily
M2 June, 15 Provide Find Usages numbers for selected projects => do that daily


Risks and dependencies

Depend on accuracy measurement results. Results of test runs must be stable and without fail alarms. Tests run slowly and to provide results daily needs number of configured machines (in progress)

Status and test results

Projects for accuracy measurement

We are using different projects to cover real usage of C/C++

  • C++ Projects
Not intensive macro usages Intensive macro usages
Not intensive template usages litesql (98.8%, templ 0.2%) (8900 CP), Quote (100%)(700 CP) Freeway (mixed a little) (99.5) (4200 CP), Povray (99.8%, templ 0.1%) (13 000 CP)
Intensive template usages loki (98%, templ 1.5%) (a little macros) (7200 CP) CLucene (99.2%, templ 1.5%) (7 000 CP), boost (96.6%, templ 1.7%) (163000 CP)
  • C Projects
Not intensive macro usages Intensive macro usages
TBD Perl (97.5%) (10500 CP), Python (97%) (12 000 CP ), OS.UTS (97.2%)
  • Mixed project (C/C++)
    1. MySQL (99.3%, templ 0.2%) (650 000 CP) - lot of templates and lot of macros
    2. ddd (99%) (*170 000 CP) - few templates and lot of macros


Design secifications

Secifications

Unit tests

Issuezilla

Feature IZ link

Status Whiteboard keyword: model_accuracy

(click on the keyword above to get issues list)



Language Model Accuracy in NB 6.5

Goal

The goal is to have 95% accuracy of Language Model.

Status

The current status: TBD (need the number)

Metrics

User features measured should have:

  • Code Completion (95% accurate)
  • Find Usages (95% accurate)

Note: projects are compiled and correctly linked

Selected Projects

We are using different projects to cover real usage of C/C++

  • C++ Projects
Not intensive macro usages Intensive macro usages
Not intensive template usages litesql(83.62%), Quote (97.36%) Freeway (mixed a little) (75.41%)
Intensive template usages loki (a little macros) CLucene(83.59%), mico
  • C Projects
Not intensive macro usages Intensive macro usages
TBD Perl, Python
  • Mixed project (C/C++)
    1. MySQL - lot of templates and lot of macros
    2. ddd - few templates and lot of macros

Projects's configuring:

  • Freeway (Done)
  • Quote (Done)
  • loki (Done)
  • litesql (Done)
  • CLucene (Done)
  • MySQL (Done ?)
  • ddd (Done)
  • mico (in progress...)
  • Perl (in progress...)
  • Python (in progress...)

Plan

  • Define projects based on criteria => fill up tables
  • Provide Code Completion Numbers for selected projects => do that daily
  • Choose golden data provider for Find Usages (may be different for C and C++ projects) => Generate golden files
  • Provide Find Usages numbers for selected projects => do that daily
  • Analyze numbers and failures to detect problem areas => create IZs and fix them

Status

Projects's configuring:

  • Freeway (Done)
  • Quote (Done)
  • loki (Done)
  • litesql (Done)
  • CLucene (Done)
  • MySQL (Done)
  • ddd (Done)
  • mico (in progress...)
  • Perl (in progress...)
  • Python (in progress...)

Code Completion

Tests running:

  • Freeway (Daily Manual)
  • Quote (Daily Manual)
  • loki (Manual)
  • litesql (Daily Manual)
  • CLucene (Daily Manual)
  • MySQL (Manual)
  • ddd (Manual)

Find Usages

Tests creation in progress...


Team

Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo