TS 74 Profiler Sanity

Profiler Test Specification - Sanity

Author: Emanuel Hucka, Alexander Kouznetsov, Matus Dekanek, Petr Cyhelsky, Jiri Skrivanek
Version: NetBeans 7.4
Last update: 19-th July 2013
Bugzilla: profiler/Base
Introduction: This test specification contains most important parts of profiler test specification.

Contents


Test suite: Installation

  1. Installation
    1. Install current version of NetBeans IDE with Java EE, GlassFish and Tomcat servers. Use a folder containing spaces to verify fixes of SPACE_IN_PATH issues.
    2. Start NetBeans IDE and check Profiler main menu is available
    3. Check Tools -> Options dialog contains Profiler tab in Java section.
    4. Invoke Profiler -> Advanced Commands -> Run Profiler Calibration
    5. Select default Java platform.
    6. Calibration should be finished by information dialog
    7. Click Show Details and see measured values
    8. Click OK

Test suite: Java SE Project

  1. Monitoring
    1. Open new project wizard and create sample project Samples -> Java -> Anagram Game
    2. Press Profile Project in main window toolbar
    3. Select Monitor in Profile AnagramGame dialog, check Enable Threads Monitoring and Enable lock contention monitoring option and press Run button.
    4. Check if Threads and VM Telemetry Overview windows are opened. Check if both windows are scrolling automatically their contents.
    5. There should not be any thread in threads window with 'Profiler' in its name.
    6. Select a thread in Threads view and invoke Thread details action in its popup menu.
    7. Check if Details tab in Threads window is selected and if it contains thread's name, 'pie chart' with statistics and line graph with colored thread's states.
    8. Select one thread's state section in the line graph. Check if Details sub tab with textual states information is opened and appropriate state is selected.
    9. Invoke VM Telemetry button in Profiler Control Panel.
    10. Press Zoom Out button (-) until whole Memory (Heap) graph panel is visible. Its horizontal scroll bar should be disabled but visible to prevent graph bumping on resize. Check if the zoomed out graph is alive.
    11. Switch to Memory (GC) graph panel and press Scale to Fit button. Whole graph should be scaled into graphs' boundaries. Wait a moment - graph should be alive and it should be shrunk to the panel boundaries - horizontal scrollbar should be disabled. The Scale to Fit button should be replaced by Fixed Scale button.
    12. Switch to Threads / Loaded Classes graph panel and check it is live
    13. Open Threads window. Go To Profiled Application and close it.
    14. Go back to profiler. There should be shown Information dialog 'The profiled application has finished ...'.
    15. Confirm the dialog and check the Threads window. There should not be any thread with 'Profiler' in its name or any new thread created during finishing of application.
    16. Profiled application should be stopped. All graphs should be stopped too.
    17. Invoke Save Current View to Image button in any graph panel. There should be opened a Select File or Directory with file system structure, preview of the view and PNG in Files of Type. Write a filename and confirm the dialog. Snapshot of selected view should be stored on disk as png file.
    18. Switch to VM Telemetry and click Export to.... Save in all possible formats and check that the created files are in correct format.
    19. Close Profiler panel. Invoke Window -> Profiling -> Profiler and Profiler panel should be opened again.
  2. Analyze Performance
    1. Invoke Profile -> Profile Project (AnagramGame) from main menu.
    2. Select CPU, Analyze Performance task, set Advanced (instrumented) and Filter: Profile Only Project Classes. Press Run button.
    3. Wait until the application is started. Open Live Results window.
    4. There should be a table with measured methods in the Live Results window.
    5. Click DrillDown button (top-right corner pie-chart button) to open DrillDown window. It should be bound to the right side of live results panel.
    6. Invoke Profile -> Advanced Commands -> View Command-line Arguments. JVM and Main class Command-line Arguments window should be opened and contain corresponding information. Close Command-line Arguments window.
    7. Invoke Stop (Terminate) the Profiled Application action.
  3. Analyze Performance - Snapshots
    1. Go to Profiler tab and invoke Rerun Last Profiling Session
    2. Check Take Snapshot button - it should be enabled during starting of the application. Press the button immediately after it is enabled.
    3. Wait until the application is fully started and close it.
    4. There should appear "Application Finished" dialog asking if you want to take a snapshot of the finished application. Click Yes button.
    5. Check if both taken snapshots contain measured data. The second snapshot should contain more measured methods than the first.
    6. Go to the first snapshot and try to close it. It should ask if you want to save the snapshot. Press No button.
    7. In the remaining snapshot window invoke Save Snapshot to Project from toolbar, close it and open it again from Saved Snapshots list in Profiler panel.
    8. Press Ctrl+F to open find dialog, write e.g. "show" into it and confirm. Try to jump through all occurrences of the word by pressing F3.
    9. Try to change View from Methods to Classes or Packages - the results tree should be changed by appropriate way. Return back to Methods view.
    10. Check sorting of the results tables by each column - press the column title bar.
    11. Invoke Save Current View to Image button in the snapshot's toolbar. There should be opened a Select File or Directory with file system structure, preview of the view and 'PNG' in Files of Type combo. Write a filename and confirm the dialog. Snapshot of selected view should be stored on disk as png file.
    12. Go to e.g. "java.awt.Component.show(boolean)", invoke popup menu on it and press Show Back Traces action. It should switch to "Back Traces for: show" tab.
    13. Select Combined tab of the snapshot window, try to select methods in both parts of split panel and check if there is always selected counterpart method in the other panel.
    14. Go to Hot Spots tab, find "com.toy.anagrams.ui.Anagrams.main" and double click on the method. It should jump into Java editor with Anagrams.java to the selected method's code.
    15. Browse Info tab panel of the snapshot. Try editing user comments, see if they persist.
    16. Close snapshot.
    17. Select the snapshot in Saved Snapshots list of Profiler panel, rename the snapshot using the Rename button, press Save As... button, choose a path and file name and save the snapshot there.
    18. Delete the snapshot by Delete button.
    19. Invoke Profile -> Load Snapshot... action from main menu, select the previous file and confirm file dialog. The snapshot should be loaded.
  4. Analyze Performance - Root Methods
    1. Right-click AnagramGame project node and choose Profile. Select CPU -> Analyze Performance, set Advanced (instrumented) and Filter: Profile All Classes. Press Run button. Wait until the application is started. Open Live Results window.
    2. Click Guess button in the application. Go to live results table, find the "com.toy.anagrams.ui.Anagrams.guessedWordActionPerformed(java.awt.event.ActionEvent)" method, invoke its popup menu and select Add to Instrumentation Roots.... In the opened "Select Settings Configuration" dialog select Analyze Performance item and click OK button.
    3. Invoke Profile -> Stop action and wait until the profiler is finished. Invoke Profile -> Profile Project... action.
    4. Choose Advanced (instrumented). Next to the radio button should be text "custom profiling roots" and "edit..." link.
    5. Click edit... link. Check if there is "com.toy.anagrams.ui.Anagrams.guessedWordActionPerformed(java.awt.event.ActionEvent)" method selected in "Edit Profiling Roots" dialog.
    6. Check check box for "nextTrialActionPerformed(java.awt.event.ActionEvent)" method and click OK button.
    7. Click Run button. Wait until the application is started and open Live Results window.
    8. Look at "Basic Telemetry" part of Profiler panel and remember number of instrumented methods. Press Guess button in the profiled application and watch the Live Results and Basic Telemetry. The number of instrumented methods should increase. Press New Word button in the application - another methods should be measured/instrumented
    9. Invoke Reset Collected Results. Live Results window should be emptied. Switch to the profiled application window - no methods should be shown in Live Results window.
    10. Invoke File -> About action in the profiled application - live results should not be changed.
    11. Invoke Profile -> Advanced Commands -> Display Internal Statistics. "Internal Statistics of Instrumentation and Hotswapping Operations" window should be opened and contain corresponding information. Push Close button.
    12. Open StaticWordLibrary.java, right-click getWord method and choose Profiling -> Add As Profiling Root Method.... Click OK button.
    13. Right-click again getWord method and choose Profiling -> Select Profiling Root Methods.... Click OK button to select configuration and check that getWord method is selected. Click Cancel.
    14. Click Modify Profiling Session... and then edit... link.
    15. Click Advanced... button and remove all methods. Click OK in both dialogs and check "default profiling roots" label is displayed in remaining dialog.
    16. Click Run and when updated stop profiling.
  5. Analyze Memory Usage
    1. Invoke Profile -> Profile Project....
    2. Select Memory button and leave default Quick (sampled) mode.
    3. Click Run button, open "Live Results" and wait until the application is started.
    4. Press Reset Collected Results button - Live Results should get empty and refreshed after couple seconds with new sample.
    5. Stop profiling.
  6. Analyze Memory Usage - Snapshots
    1. Invoke Profile -> Profile Project... action. Select Memory -> Analyze Memory -> Advanced (Instrumented).
    2. Select Record Stack Trace for Allocations. Confirm dialog by Run button.
    3. Check Take Snapshot is enabled during starting of the application.
    4. Press Take Snapshot immediately after it is enabled.
    5. Click Export to... and save it.
    6. Close snapshot.
    7. Choose "java.util.HashMap" and invoke Take Snapshot and Show Allocation Stack Traces from its popup menu. A new snapshot should be opened with selected "Allocation Stack Traces" tab panel.
    8. Close snapshot without saving it.
    9. Close the profiled application and check that appears "Application Finished" dialog asking if you want to take snapshot of finished application. Click Yes.
    10. Click Save Snapshot to Project, close it and open it again from "Saved Snapshots" list.
    11. Press Ctrl+F, write "String" and click "Find". Find next occurrence (F3).
    12. Right-click "java.lang.String" class and choose Show Allocation Stack Traces.
    13. Press Show or hide results column button in the right-top corner, uncheck Bytes Allocated, Objects Allocated, Filter items. Press the button again and enable all items.
    14. Invoke Save Current View to Image button in the snapshot's toolbar. Write a file name and confirm the dialog.
    15. Switch to Info tab of the snapshot and check its content.
    16. Select the snapshot in "Saved Snapshots", press Save As... button, choose a path and file name and save it.
    17. Delete the snapshot by Delete button.
    18. Invoke main menu Profile -> Load Snapshot, open previously stored snapshot and then close it.
    19. In main menu choose Profile -> Compare Snapshots and browse for two previously saved snapshots.
    20. Check plus and minus values in opened "Allocations Comparison" panel.
    21. Close panel.

Test suite: NB Module

  1. Analyze Performance
    1. Create sample project File -> New Project -> Samples -> NetBeans Modules -> Paint Application. Try to start project by Run action and check if all works.
    2. Invoke main menu Profile -> Profile Project... action.
    3. Select CPU and click Create Custom... link.
    4. In New Custom Configuration (CPU) dialog change its name to "testconfig" and click OK.
    5. Select Advanced (instruments) and click Advanced Settings link.
    6. Select Exact call tree, sampled timing.
    7. Execute profiling by Run button and watch Take Snapshot button. Immediately after the button is enabled, press it. It should take non empty CPU snapshot - close it without saving. Wait until whole application is started.
    8. In the started application open New Canvas and paint something there. Close the application and confirm "Application finished dialog" in profiler by Yes button.
    9. In the taken snapshot go to Hot Spots, find method "org.netbeans.paint.PaintCanvas.mouseDragged", invoke its popup menu and select Add to Instrumentation Roots. In "Select Settings Configuration" dialog select "testconfig" configuration and confirm it by OK button.
    10. Right-click "Paint Application" project node and choose Profile. Leave 'testconfig' custom profiling selected, click Edit link.
    11. In "Edit Profiling Roots" deselect the method mouseDragged and select mouseClicked() method. Click OK.
    12. Click Run button. Wait until the application is started.
    13. Open "Live Results" window - it should be empty. Go to the profiled application, create New Canvas and try to draw something. Method mouseClicked should be displayed in Live Results window.
    14. Right-click org.netbeans.paint.PaintCanvas.mouseClicked(java.awt.event.MouseEvent) method and choose Go To Source. There should be opened editor with the selected class and its cursor should be moved to the selected method.
    15. Invoke Take Snapshot action.
    16. Invoke Profile -> Stop Profiling Session.
  2. Monitor and lock contention
    1. Invoke Profile -> Profile Project (Paint Application) from main menu.
    2. Select Monitor -> Monitor Application, select Enable threads monitoring, Enable lock contention monitoring and press Run button.
    3. Wait until the application is started and open New Canvas there and check Threads window if it is working.
    4. Switch to "Lock Contention" window.
    5. Under each thread is set of objects/locks on which synchronization was done. It may signal bottlenecks when single object is locked many times and with different threads.
    6. Try to export this view as image or other file type.
    7. Select Monitors in "View by:" combo box and you should see list of objects.
    8. Expand several nodes and check threads accessing object are listed underneath.
  3. Analyze Memory Usage
    1. Invoke Modify Profiling Session... action, select Memory -> Analyze Memory -> Advanced, select both Record full object lifecycle and Record stack trace for allocations. Click Run to confirm.
    2. Play with application to produce some profiling results.
    3. Open Live Results window, right-click java.awt.event.MouseEvent class and choose Log Class History.
    4. Click Scale to Fit in opened History of java.awt.event.MouseEvent.
    5. Go to the profiled application and draw something.
    6. Check both graphs increased evidently.
    7. Click Run Garbage Collection in Profiled Application and check "Live Objects" and "Live Bytes" decrease to zero.
    8. Stop profiling.

Test suite: Web Project

  1. Monitoring
    1. Open new project wizard and create "Java Web|Web Application" project (Java EE 7, GlassFish Server).
    2. Right-click project node and choose Profile.
    3. Select Monitor -> Monitor Application in Profile WebApplication dialog, select Enable threads monitoring option and press Run button.
    4. Check if Threads and VM Telemetry Overview windows are opened. Check if both windows are scrolling automatically their contents.
    5. There should not be any thread in threads window with Profiler or JFluid in its name.
    6. Right-click a thread in Threads view and choose Thread Details.
    7. Check if Details tab in Threads window is selected and if it contains thread's name, pie chart with statistics and line graph with colored thread's states.
    8. Select one thread's state section in line graph. Check if Details sub tab with textual states information is opened and appropriate state is selected.
    9. Invoke VM Telemetry button in Profiler control panel.
    10. Press Zoom Out button (-) until whole Memory (Heap) graph panel is visible. Its horizontal scroll bar should be disabled but visible to prevent graph bumping on resize.
    11. Switch to "Memory (GC)" graph panel and press Scale to Fit button. Whole graph should be scaled into graphs' boundaries - horizontal scroll bar should be disabled. The "Scale to Fit" button should be replaced by "Fixed Scale" button.
    12. Switch to "Threads/Loaded Classes" graph panel and use mouse wheel to zoom out and in.
    13. Open Threads window. Go To Services view and stop GlassFish.
    14. Information dialog The profiled application has finished ... is opened. Wait a few seconds - Threads window on background should continue in showing new states of threads.
    15. Confirm the dialog and check the Threads window is stopped.
    16. Invoke Save Current View to Image button in any graph panel and try to save it.

Test suite: Attach Profiler

  1. Local Dynamic
    1. Run AnagramGame project
    2. Invoke Profile -> Attach Profiler... from main menu.
    3. Select "AnagramGame" in top combo box Attach to:.
    4. Select CPU -> Analyze Performance -> Advanced -> Profile only project classes.
    5. If you are attaching profiler for the first time, "Attach Mode" is not yet defined - click define.... Otherwise click change... at the bottom of "Attach Profiler" dialog and select Local and Dynamic type.
    6. Click Attach.
    7. Select Anagrams process and click OK.
    8. Open "Live Results", click something in profiled application and check methods from AnagramGame project are displayed.
    9. Click Detach from the Profiled Application in Profiler panel.
    10. Click Yes in "Detach Profiler" window and check profiler is stopped and profiled application is terminated.

Test suite: HeapWalker

  1. Summary
    1. Right-click project node and choose Profile
    2. Select CPU -> Advanced -> Profile only project classes and click Run.
    3. Open "Live Results" and wait until application is started.
    4. Click Dump Heap, choose destination to "Custom Directory" and click No to not open it in HeapWalker.
    5. Click something several times in profiled application.
    6. Click Dump Heap again, choose destination to "Profiled Project" and click Yes to open it in HeapWalker.
    7. Check Summary view contains overall data. Click Show System Properties and Show Threads. Check links in stack traces are valid.
    8. Click link in "Application windows" and return back to Summary panel.
    9. Click Find to search for biggest objects by retained size. Confirm question dialog and when finished click some links to get to Instances view.
    10. Close heap dump and open it again from "Saved Snapshots".
  2. Classes
    1. Switch to Classes page.
    2. Check sorting of table by each column - press the column title bar.
    3. Press Show or hide columns small button in the right-top corner of the table, uncheck Instances, Size, Filter items. There should remain only two columns in the table. Press the button again and enable all items in the menu.
    4. Go to Class Name Filter (bottom line of panel) and select Contains filter type. Write Buffer and press Enter. Classes table should contain only 'Buffers'. Press Clear filter button with red cross on the filter line. Table should contain all classes again. Try to select e.g. Regular expression filter type - write .*Map to filter text field and press Apply filter button - there should be shown all 'Maps'. Press Clear filter button.
    5. Select java.lang.String class and press Static Fields button (top-right corner of window). Static Fields panel should be inserted to the right side of heapdump window. There should be shown all static fields of String class (class, serialVersionUID etc.). There should be shown icons legend in the bottom line of the window (Array Type, Object Type etc.)
    6. Right-click java.lang.String and choose Go to Source. It should open source file of java.lang.String class in editor and move cursor to class definition. (It doesn't work on some JDKs on mac os x).
    7. Click Compare with another heap dump link at the top right of Classes tab. Browse for heap dump previously stored at custom folder.
    8. Review plus and minus values across table and click reset view.
  3. Instances
    1. Right-click java.lang.String and choose Show in Instances View.
    2. Try to close and open again all panels in Instances view. Use 'x' buttons in right-top corner of each panel and buttons with icons on heap dump window toolbar. Try to change size of panels - after closing and opening again the size of panel should be preserved.
    3. Check the legend line bellow the toolbar - there should be name of class and some statistics about it.
    4. Try to switch on and off all available columns in each panel by Show or hide columns button (top-right corner of each panel).
    5. Try to sort instances list table by particular columns - press title of the column.
    6. Find a table cell (mostly in Fields or References panels) which's content is not fully displayed (ends by ...). Move mouse cursor over the cell and check if its full content is shown in appeared tooltip.
    7. Click on the retained size link at the top right - the retained sizes should be computed and displayed in new column.
    8. Sort instances by retained size and select first item. Check there is value partially shown in description panel. Click Show all to see entire value. Click Save to file and save value to disk.
    9. Right-click a node of non-primitive type in Fields or References panel and choose Show Instance. Instances of the class should be displayed in heap dump window and there should be selected the instance in Instances list panel.
    10. Click Navigate back arrow in toolbar.
    11. Right-click a node of non-primitive type in Fields panel and choose Show in Classes View. Classes view should be displayed in heap dump window with the selected class. Return back to Instances.
    12. Right-click a node of non-primitive type in Fields or References panel and choose Go to Source. The editor with source of the class should be opened.
    13. Right-click this field in References view and choose Show Nearest GC Root.
    14. Right-click select GC root and choose Copy Path From Root. Check that the path is copied into clipboard.

Test suite: Profiling Points

  1. Stopwatch and Reset Results
    1. Open "Anagrams.java in editor. Move cursor to the first line of method guessedWordActionPerformed (there is if command).
    2. Invoke popup menu there and press Profiling -> Insert Profiling Point menu item.
    3. Select "Profiling Point Type" Stopwatch.
    4. Click Next and confirm default settings by pressing Finish button.
    5. The stopwatch annotation should be added to the line, line should have different background color.
    6. Invoke popup menu on the annotation, there should be Stopwatch Profiling Point sub menu containing Enabled, Edit, Delete and Go To End Point options. Select Go To End Point. Warning message is shown.
    7. Again open popup and now select Profiling Point > Edit menu item.
    8. Choose Timestamp and duration radio button. Location (stop) controls gets enabled. Set line number to 274 (last line of method guessedWordActionPerformed). Click OK.
    9. Invoke popup menu on the end stopwatch annotation (line 274), select Profiling Point -> Go To Start Point. Cursor should be moved to the start stopwatch line.
    10. Invoke popup menu on the start stopwatch annotation (line 266), select Profiling Point -> Go To End Point. Cursor should be moved to the end stopwatch line.
    11. Move cursor to the first line of method nextTrialActionPerformed. Press Shift-F9. Reset Results profiling point is inserted and Customize Profiling Point dialog is shown. Confirm it by OK button.
    12. Right-click AnagramGame project node and choose Profile.
    13. Select CPU - > Analyze Performance -> Profile Only Project Classes. Ensure that Use defined Profiling Points option is enabled and Show active Profiling Points link opens list with two Profiling Points.
    14. Press Run button and wait until the application is started.
    15. Press its Guess button five times. "Profiling Points" window should show 5 hits for Stopwatch profiling point and no results available for reset results profiling point.
    16. Click report link for Stopwatch profiling point. Report window should be shown, for each hit timestamp and duration should be shown.
    17. Open Live Results. There should be some measured methods. Click New Word button in profiled application. Live Results should show only two methods from now.
    18. Click report link for Reset Results in Profiling Points window. Report window should be shown, for each hit timestamp should be shown.
    19. Stop Profiling.
    20. Start Profiling again having User defined Profiling Points option disabled. Play with application. No results for profiling points should be shown. Stop Profiling.
    21. Right-click Reset Results in Profiling Points window and select Show in Source menu item. Cursor should be moved to first line of nextTrialActionPerformed method.
    22. Rigth-click its annotation and select Profiling Points > Enabled. Reset Results Profiling point should be shown gray in editor and Profiling Points window.
    23. Right-click Stopwatch in Profiling Points window and select Show Start in Source menu item. Cursor should be moved to first line of guessedWordActionPerformed method. Right-click Stopwatch in Profiling Points window and select Show End in Source menu item. Cursor should be moved to last line of guessedWordActionPerformed method.
    24. Rigth-click its annotation and select Profiling Points > Delete. Stopwatch Profiling point should be removed from editor and Profiling Points window.
Not logged in. Log in, Register

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo