Add synchronization with external systems like bug trackers

[rsenden]

Rationale

We currently provide FortifyBugTrackerUtility (FBTU) for submitting vulnerability data to bug tracking and other external systems. It makes sense to move this functionality to FortifyVulnerabilityExporter (FVE) for various reasons:

• Submitting vulnerability data to a bug tracking system is just a special kind of export
• There is already some overlap between the utilities, for example both can export vulnerability data to CSV format
• Despite its name, FBTU supports exporting vulnerability data to non-bugtracking systems like CSV files and RSA Archer; having a generic export utility for such non-bugtracking targets makes more sense
• FVE provides a much more modern and user-friendly configuration file format (YAML versus Spring XML)
• FVE has a much simpler architecture and code base, utilizing modern Java features like lambda expressions
• FVE build system and automated semantic versioning make it easier to publish small updates as patches

Requirements

Compatibility with FBTU is the main requirement for moving bug tracking support to FVE:

• Bug links on vulnerabilities previously exported with FBTU must be properly recognized and processed by FVE
  • Bug links stored as native bug links in SSC or FoD
  • Bug links stored as comments in FoD
  • Bug links stored as custom tags in SSC
• Support for updating SSC custom tags with current (bug) state in target system
• Support for updating previously submitted bugs/work items with updated SSC/FoD vulnerability state
  • Update bug/work item contents like description
  • Transition bug/work item to open/closed status
• Support for the following external (bug tracking) systems:
  • ALM Octane: required
  • JIRA: required, including support for parent issues
  • Azure DevOps: required
  • CSV: already provided by FVE
  • RSA Archer: not planned unless there is demand
  • SSC bug trackers: not planned unless there is demand

Ideas for implementation

Vulnerability loading

• Contrary to FBTU, FVE will by default load all vulnerabilities from FoD/SSC and process them all at once for both submitting new vulnerabilities and updating existing bugs/work items
  • Avoids loading vulnerability data twice in case we need to do client-side filtering to differentiate between previously submitted vulnerabilities and new vulnerabilities to be exported
  • Allows for easier, explicit, flexible and less error-prone configuration using client-side filtering
    • Explicit client-side filters specify whether hidden, removed or suppressed issues should be exported
    • Example filter expression for submitting new vulnerabilities: isOpen && !targetLocation && customTags['Analysis']=='Exploitable'
  • Client-side filters are specified on the target configuration, rather than SSC/FoD configuration
    • Allows target implementation to provide different filtering options for different purposes
      • For example, bug tracker implementation can provide different filtering properties for handling new versus previously submitted vulnerabilities
    • Allows for exporting vulnerabilities to multiple systems/formats in a single run

Enhance embed processing

• Add expression property to fortify-client-api StandardEmbedConfig class

  • Provides explicit alternative for FBTU isVulnerabilityOpenExpression, by configuring for example ssc.vulnerability.embed with propertyName: isOpen and expression: removed==false && suppressed==false && hidden==false

• Add support in fortify-client-api to embed SSC custom tag values by name

  • Allows for filtering by for example Analysis tag value
  • Allows for easy retrieval of export location from custom tag

Storing export details in SSC/FoD

• Add configuration properties for FoD/SSC updates
  • <targetType.instanceName>.<ssc|fod>.store:
    • exportLocation:
      • asNativeBugLink: <true|false> (SSC & FoD)
      • asCustomTag: <name> (SSC)
      • asComment: <commentFormat> (FoD)
    • extraCustomTags: (SSC)
      • <name>: <expression>
  • Although property is under target configuration, it will be read by SSC/FoD implementation
    • For each configured target factory, SSC/FoD implementation will get <targetType.instanceName> from the factory and read the corresponding <targetType.instanceName>.<ssc|fod>.store: configuration
    • If <targetType.instanceName>.<ssc|fod>.store.exportLocation.asNativeBugLink is true, the target processor will not be created/invoked if native bug link is not available for the current application version/release (incorrect bug tracker configured on FoD, 'Register bug link' bug tracker not configured on SSC)
    • If <targetType.instanceName>.ssc.store.exportLocation.asCustomTag is defined, the target processor will not be created/invoked if custom tag is not available on current application version
    • If the application version/release is skipped for all configured targets, effectively no vulnerabilities will be loaded for that application version/release
• Add updateVulnerabilityState method in IVulnerabilityConsumerContext that can be invoked by target implementation to request an update of SSC/FoD vulnerability state, for example to register export location (for newly exported issues) or to update extra custom tags in SSC (for newly and previously exported issues)
  • Target passes <targetType.instanceName>, list of vulnerabilities (contents or id's?) to be updated, and target entity details (location, state, ...) to this method
  • Is it necessary to differentiate between newly exported issues and previously exported issues?
  • If so, separate methods or enum value?

Vulnerability grouping

• Grouping is configured and performed by the target implementation
  • Allows for different groupings for different targets in a single run
• Standard grouping functionality is provided for easy re-use by the target implementations
  • Can we re-use/generalize existing JSON formatting functionality to also perform grouping?