Malcolm v6.0.0

Malcolm v6.0.0 is a major release which incorporates Suricata as a data source for network traffic analysis in Malcolm alongside Zeek and Arkime. A team at BYU (@piercema, @aglad-eng, @Jarscott1, @n8hacks) recently completed their work on Suricata integration for their capstone project. This release includes their changes as well as some additional work by Malcolm's developer in integrating Suricata in other ways not covered in the scope of their project. This release also includes other bug fixes and improvements.

v5.2.11...v6.0.0

As the Malcolm project uses semantic versioning when choosing version numbers. This release required some pretty extensive remapping of Zeek fields in order for Zeek and Suricata to target the same naming conventions for common fields. This backwards-compatibility breaking change is the reason for bumping the major version number from 5 to 6. It is not recommended to attempt an upgrade from a previous release; a fresh install is strongly encouraged.

• Features

  • Incorporate Suricata as a data source for network traffic analysis in both Malcolm and Hedgehog Linux
  • Added support for the GENISYS protocol

• Improvements

  • Minor tweaks to the GitHub workflows for building the Malcolm installer ISO
  • Better fingerprinting of events during Logstash parsing in order to create a unique but reproducible hash for events in the case that duplicate data is indexed into Malcolm
  • All data sources (Arkime, Zeek and Suricata) now specify the data source (stored as event.provider, arkime, zeek and suricata, respectively) and the log type (stored as event.dataset, e.g., session, conn, alert, etc.) in order to facilitate filtering among various types of network metadata
  • The Malcolm REST API was improved to support POST operations for all of the calls which can accept a filter argument to allow for easier representation of filters as JSON objects
  • Reworked several dashboards, including the Overview, Security Overview, Zeek Notices and Signatures dashboards
  • Leave packages in place on the ISO-installed Malcolm and Hedgehog Linux environments in order to support mounting SMB shares from the Thunar GUI

• Bug fixes

  • Fix idaholab#94: docker-compose | "function" has no attribute "get" (ubuntu 20.04 install)
  • Fix idaholab#96: DNP3 dashboard has invalid saved search syntax
  • Fix idaholab#97: virustotal file scanning broken (AttributeError: 'Namespace' object has no attribute 'vtotReqLimit')
  • Fix idaholab#98: BSAP RDB data parsed incorrectly

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.11

Malcolm v5.2.11 is a minor release with a few user experience improvements and component version updates (some of which resolve potential security issues).

v5.2.10...v5.2.11

• Addressing security vulnerabilities

  • bump Zeek to v4.2.1 addressing a potential Zeek buffer overflow vulnerability
  • Deserilization of Untrusted YML data - #207

• Version bumps

  • Spicy to v1.4.1
  • OpenSearch to v1.3.1
  • Yara to v4.2.1

• Improvements

  • Resolve performance degredation when we went to OpenSearch 1.3 by using the G1GC garbage collector - idaholab#91
  • improve workflow for configuring Malcolm to run behind another reverse proxy (Caddy, Traefik, etc.) - idaholab#92
  • assign and display both event.provider and event.dataset in Arkime - idaholab#89
  • only show the controls for PCAP download from session details if there is actually a PCAP backing the session document #90 - idaholab#90
  • increase timeouts related to filebeat (see https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html) to be a little more forgiving for log files that take a long time to process - mmguero-dev/Malcolm@04b6084
  • strip build status badges from deployed copy of README.md
  • The install.py script will make use of the pythondialog module for user interaction (on Linux) if it is available
  • added link to Dashboards in the footer of Arkime's interface

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.10

Malcolm v5.2.10 is a minor release updating some of Malcolm's core components.

v5.2.9...v5.2.10

• Version bumps

  • Arkime to v3.4.2
  • OpenSearch to v1.3.0
  • OpenSearch dashboards to v1.3.0

• Bug fixes

  • #205
  • ensure timestamp fields are explicitly defined as date type in index template

• Improvements

  • restore zeek.cip_io.io_data field so that it may be reviewed in Dashboards Discover view and Arkime
  • added malcolmmonitor convenience bash function into Malcolm ISO-installed environment
  • pointed several zeek plugins' installation source back upstream now that my PRs have been accepted

• Cleanup

  • removed references related to internally-developed INL tool MALASS which is no longer under development and was never released publicly

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.9

Malcolm v5.2.9 is a release to fix a regression introduced in v5.2.9 (see idaholab#84), affecting the Malcolm REST API and generation of intelligence files for Zeek. If you don't use those features, you may choose to skip this release. My apologies for putting this out so close to the last release.

v5.2.8...v5.2.9

• Bug fixes
  • Fix idaholab#84 ("upstream incompatibility between python regex library 2022.3.15 and dateparser breaks API")

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.8

Malcolm v5.2.8 is a release to patch a major security vulnerability in OpenSSL.

v5.2.7...v5.2.8

• Version bumps

  • Arkime to v3.4.1
  • Spicy to v1.4.0
  • Update all docker images' system packages to get latest security updates, including updating OpenSSL to fix CVE-2022-0778
    • CVE-2022-0778 can already be detected in network traffic by Malcolm by 0xxon/cve-2020-0601

• Minor improvements

  • Include gvfs-backends package in ISO-installed environments to allow mounting SMB shares in the Thunar GUI

• Bug fixes

  • Fix an issue with "read-only mode" combined with "no SSL mode" (very unlikely to have affected anybody)
  • Tweak Logstash pipeline size to make it a little more conservative to avoid Logstash restarts due to running out of heap resources

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.7

Malcolm v5.2.7 is a patch release with improvements and bug fixes.

v5.2.6...v5.2.7

• Bugs fixed

  • fixed instances where spicy_ will sometimes be prepended to network.protocol fields (e.g., spicy_wireguard is now fixed to just be wireguard)

• Improvements

  • base GitHub workflow files' docker build step on moby/buildkit:master
  • added API webhook that can be used as an Alerting destionation for alerts to be indexed back into the OpenSearch database as session records
  • added example Alerting monitor and destination using API webhook
  • added ability to run Malcolm's nginx-proxy container in non-HTTPs mode (not recommended unless running behind a third-party reverse proxy like Traefik or Caddy, in which case it is very useful)
  • removed performance-analyzer plugin from OpenSearch container to free up resources
  • improvements to documentation for Anomaly Detection and Alerting
  • added example scripts and Vagrantfile for easily configuring and running Malcolm in a read-only or demo mode on Amazon Linux 2 (useful for AWS)

• Version bumps

  • Arkime to v3.4.0
  • Yara to v4.2.0
  • Capa to v3.2.0

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.6

Malcolm v5.2.6 is a patch release with improvements and bug fixes.

v5.2.5...v5.2.6

• Bugs fixed

  • Fixed Logstash failing to start idaholab#78
  • Added tuning options to address Logstash out of memory errors idaholab#79
  • Incorporated latest bugfixes in BACnet parser
  • Fixed issue with mapping some field types being incorrect for BSAP and OSPF logs

• Improvements

  • Added http-more-files-names plugin to populate files.log filenames entries for HTTP requests
  • Normalized bsap_ip_header.type_name to event.action
  • Removed unnecessary Logstash field conversions for types already defined in the template
  • Improved logs and status convenience scripts to allow filtering to a particular service
  • Improved convenience script for working with GitHub workflows during Malcolm development

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.5

Malcolm v5.2.5 is a patch release with improvements and bug fixes.

v5.2.4...v5.2.5

• Threat Intelligence

  • idaholab#77 - automatically generate Zeek intelligence indicators from MISP
  • perform autogeneration of Zeek intel files from TAXII/MISP feeds multithreaded
  • allow filtering indicators from TAXII/MISP by date (e.g., "only include those created/modified in the last n days", etc.)
  • added intelligence hits as a new severity ranked category
  • highlight intel sources more clearly in dashboard

• Hedgehog Linux (sensor appliance)

  • added sensormonitor convenience function to monitor services, disk space and logs

• Bug fixes

  • Remove CIP fields no longer supplied by the ICSNPP EtherNet/IP parser and update dashboard accordingly
  • idaholab#76 - directory creation race condition starting up zeek on sensor which may cause zeekctl to fail
  • #189 - mount destination [/opt/zeek/share/zeek/site/intel] not absolute: unknown

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.4

Malcolm v5.2.4 is a patch release with improvements and bug fixes.

v5.2.3...v5.2.4

• New features

  • idaholab#74 (automatically generate Zeek intelligence indicators from STIX/TAXII)

• Improvements

  • group MAC addresses and OUI (vendors) into related.mac and related.oui for easier searching across all fields
  • improvements to default anomaly detectors

• Bug fixes

  • Fix idaholab#75 (OpenSearch Dashboards loads slowly without network connectivity)
  • Fix idaholab#76 (directory creation race condition starting up zeek on sensor which may cause zeekctl to fail)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.3

Malcolm v5.2.3 is a patch release with component version bumps, bug fixes and improvements.

v5.2.2...v5.2.3

• Version bumps

  • Arkime v3.3.1
  • Zeek v4.2.0

• Improvements

  • Added script and better documentation for putting Malcolm in "read-only" mode
  • Improved Files dashboard

• Bug fixes

  • Fixed an issue where Logstash wasn't parsing the ftime from files.log correctly (a field added by the Spicy ZIP analyzer)
  • Fixed idaholab#73 (path for tcpdump changed) for Hedgehog Linux
  • Fixed idaholab#72 (better file directory/name parsing and normalization in Logstash)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.