Report Exception FAQs
How the report exception works from user's point of view?
- User can click on Report Error button in Exception dialog and specify following data:
- Issuezilla login name (isn't required, when the field is empty the anonymous is used)
- Summary is going to be initially set to exception class and exception message, if there is some, but we are giving user a possibility to change this item.
- User can fill short description what has happen to him.
- All data that are sent to server would be checked in View Data dialog
- The report is sent to server when the user clicks on Submit Data button
- User is informed about report on web page that is opened next
Can we disabled reporting exceptions on server side?
Yes, reporting exceptions would be disabled on server side. We can only change HTML page and user will be asked to report exception in Issuezilla when he clicks on Report problem.
How the exception report is processed on server side?
Following checks are performed when the new exception report is received on server side:
- Create new entry in database for new exception report
- Query existing exception stack traces and check whether there are some similarities. If some similar report is found the new report is marked as duplicate of existing and user is informed about this on web page.
- Assign the Issuezilla component/subcomponent to report according to jar file where the exception was thrown
Which steps should be performed by QA engineers?
- Use unmapped reports list and select the reports that belongs to component you are responsible. Order the list by number of duplicates and the build number.
- Check the assigned component/subcomponent.
- Query IZ and find the existing issue. Type the issue number in detailed page.
- If there is no existing issue, click on enter link and create new IZ bug.
Which data are pre-defined in New IZ issue?
- priority is set to P3
- reporter is added to cc
- OS, jdk and several stack trace lines are in comment
- link to exception report URL
Do you check existing bugs in Issuezilla before saving new exception?
No, since we found out that make a query for every new error report is the performance problem for Collabnet servers.
Do you use some authorization on web pages?
Can user add a comment for existing exception report?
Yes, he can add comment when he reports exception and also he can go to exception detail page and write the comment there.
How the exceptions are transfered in bug tracking system
The reports about exceptions are sent from user's IDE in database and then the exceptions are processed on the server further. The rules for reporting exceptions in IZ are listed below:
- Reports that are sent by NetBeans QA engineers are migrated directly in IZ. Jan Horvath maintains the list.
- The exception has more then five duplicates from unique users
How the subcomponent/component is resolved from exception?
The algorithm is described here.
What's the algorithm how to mark the report as duplicate? What's the effectiveness of this algorithm?
Input of the algorithm is a new Throwable and output is an Exception object from DB of which is the Throwable duplicate of. Algorithm is made of two filters.
First filter is based on hashcodes. It counts a hashcode of the stacktrace and try to find the same hashcode in DB, if the same hashcode is found, all the lines of stacktrace are verified to be the same and Throwable is marked as a duplicate of issue assigned to hashcode in DB. If no hashcode is found or lines are different, second filter is started.
Second filter is based on lines comparision. First line of new stacktrace is found in DB and all Exceptions with the same first line are possible candidates for a duplicate. From this group of candidates Exceptions with classes (NPE, IOE, ...) different from the new one are removed - in some situations (example: ClassNotFoundException) exception message is checked as well. In next step second line of stacktrace is found in DB and the group of candidates is reduced for Exceptions with the same first two lines. The reducing continues with third, fourth, ... line until the last line of matching is found. If the number of same lines is higher then "three" (parametrical value) new exception is marked as a duplicate of this exception.
The reason for the first filter is to find exact duplicates and to do it as fast as possible thanks to a hashcode. The reason for the second filter is to find the most similar stacktrace between all the reports and decide whether it is a duplicate or not.
It's hard to judge the effectiveness of this algorithm. In some situation where the duplicity is based on stacktrace this algorithm should work well, but there are situations of duplicates where just one line of stacktrace is the same and so this algorithm fails. The actual design suppose new ideas and changes. Every one can write his own filter and it can be inserted in the filtration queue. We would also need some metric of the quality of this algorithm, but neither a human is always able to decide about duplicates, so how to measure this quality.