Ideas for future work

[gharrma]

A laundry list of ideas.

The ideas within each section are sorted very roughly by value.

Note: I generally like the idea that tools should "do one thing and do it well". So, most of these ideas are about improving the core functionality of IDE Perf (which is: tracing methods on-demand and viewing the data in real time).

New features

• Support tracing all overriding methods of some base method.
  • This is extremely useful because e.g. you can trace the methods in an IntelliJ extension point, and then compare the performance of the corresponding extensions across all plugins.
• Support loading tracing configurations from text files.
  • This will make tracing configurations persistable and shareable.
  • We could also support loading tracing configurations at startup, in order to trace methods which run early in the IDE startup sequence.
• Built-in tracing profiles for common IntelliJ classes:
  • Trace all syntax highlighting and inspection passes.
  • Trace all extension points and listeners.
  • Trace PSI resolve.
• Record CPU time too using ThreadMXBean.getCurrentThreadCpuTime.
  • Note, ThreadMXBean has high overhead (1+ microseconds), so we would need to limit this feature when tracing rapidly-called methods. Or, explore the overhead of using C/C++ timers through JNI.
• Automatically trace all frames in a copy-pasted stack trace.
• Memory allocation sampling to estimate memory allocations in traced methods.
  • See Add capability to sample allocations of class #39.
  • See Add native JVMTI agent with allocation sampling #35.
• Auto-trace interesting methods based on built-in CPU sampling.
• Adaptive tracing: auto-untrace rapidly-called methods to reduce tracing overhead.
• Capture backtraces for traced methods.
  • Need to keep runtime overhead low, though; maybe use rate-limiting.
• Misc runtime bug finders:
  • Auto-detect read actions which block the write lock for a while.
  • Auto-detect long read actions which do not call checkCanceled.
  • Auto-detect hash maps with excessive hash collisions.
  • Auto-detect duplicate string allocations.
• Support for tracing specific program points by line number.
  • This would be useful for counting loop iterations, or tracing lambdas (which are difficult to identify by name).
• Support a flag -no-merge which will trace every invocation of a method without merging the results.
• Support tracing native methods using the setNativeMethodPrefix API.

UX polish

• Make tracing auto-completion work even for unloaded classes.
• Support column sorting in the call tree view.
• Add a warning indicator if tracing overhead is too high for a given method.
• Add ability to restore the full tracing configuration from a previous IDE session.
• Allow dumping the trace data into text form---preferrably by making the tracer table copy-paste-able.
• Help new users by adding some instructions in the tracer window on how to get started.
• Add a status pane or log to indicate which methods have successfully been traced. Alternatively, show traced methods in the tracepoint table even when their call count it zero.
• Improve the auto-completion experience for anonymous classes, including lambdas.
• Dislay UI overhead as a percentage (of the last second) rather than a total.
• Add UI buttons for common actions such as 'clear' and 'reset'.
• Add a right-click menu to the tracepoint table, making it easier to untrace specific methods.
  • Support multiple selection too, so that e.g. it is easy to untrace at once all methods which are adding too much tracing overhead.
• Tracepoint details dialog:
  • Make it more discoverable.
  • Show which threads the current method was seen on.
  • Make it work for the call tree view too.
• Display total wall time across all threads under a synthetic 'root' tracepoint.
• Add color to better visualize trace data (e.g. red color for hot methods).
• Allow canceling the trace command (for when the user accidentally traces too many methods). Also cancel commands when the tracer window is closed(?).
• Parameter tracing: allow specifying parameters by name rather than position. Also consider renaming to "argument tracing".
• Try to support dynamic unloading of the IDE Perf plugin, in order to improve the plugin update flow.
  • Currently quite difficult due to classloader leaks in IntelliJ.

Optimizations, testing, refactoring, cleanup, minor bugs

• Reduce memory churn caused by CallTreeManager and its frequent tree copying. (It can lead to high GC overhead if too many methods are traced.)
  • One idea is to reuse old call tree nodes.
• Reduce UI rendering overhead, especially when the tracing data is unchanged.
• Explore ways to improve the speed of bytecode transformations on loaded classes, and/or offer a way to "restart IDE with current tracing configuration" to bypass the long wait when tracing many methods.
• Simplify tracer command parsing (TracerCommand).
• CachedValue tracer:
  • Add a proper integration test analogous to TracerIntegrationTest.
  • Consider replacing our CachedValue tracer with the upstream tool described in IDEA-261466 (only available in IntelliJ Ultimate?).
• Currently the VFS tracer uses very custom tracing hooks to gather data. Consider creating a more general framework/DSL for quickly creating custom probes.
• Consider merging the CachedValue tracer and the VFS tracer into the main tracer window.
• Improve TracerIntegrationTest:
  • Test the UI too.
  • Test traced methods throwing exceptions.
  • Add some multi-threaded stress tests.
  • Stress-test the bytecode transformer by tracing many methods (e.g. entire JDK).
  • Add a test for thread chooser.
• Technically two MethodTracepointWithArg instances should be considered different if they have different parameter indices.
• Audit computation of average wall time for recursive methods.
• Consider removing support for the -count flag, since its use cases seem limited in hindsight.
• For parameter tracing, we should wait to call toString until we are inside the recursion prevention guard.

Misc

• Create a GIF of the tracer to show in the README.
• Add more documentation on the architecture of IDE Perf.
• Write some sort of blog post showing examples of using IDE Perf to solve performance issues.