Merge pull request #297 from cisagov/v24.02.0_merge_cisagov

Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

* Features and enhancements
    - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this)
    - allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    - add option for customizing which log types get NetBox enrichment (idaholab#316)
    - improve the extracted_files download page (idaholab#329)
    - include missing aggregations in API bucket queries (idaholab#386)
    - more intelligent .env file checking on startup (idaholab#387)
    - Malcolm report to itself on capture statistics (idaholab#395)
    - link to Dashboards/Arkime from NetBox devices view (idaholab#410)
    - changed default PCAP storage format to zstd(3) for new installations
    - various documentation updates and improvements
    - changed back to using official Zeek .deb files rather than building from source to reduce build times
* Component version updates
    - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130)
    - Capa to [v7.0.1](https://github.com/mandiant/capa/releases)
    - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases)
    - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html)
    - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html)
    - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1)
* Bug fixes
    - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335)
    - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403)
    - allow netbox-restore and netbox-backup to specify container name (idaholab#337)
    - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch))
    - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401)
    - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402)
    - basic authentication breaks with special characters (idaholab#404)
    - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency))
* Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config))
    * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    ```
    # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
    # Whether or not Arkime should perform index management
    INDEX_MANAGEMENT_ENABLED=false
    # Time in hours/days before moving to warm and force merge (number followed by h or d)
    INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
    # Time in hours/days before deleting index (number followed by h or d)
    INDEX_MANAGEMENT_RETENTION_TIME=90d
    # Number of replicas for older sessions indices
    INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
    # Number of weeks of history to retain
    INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
    # Number of segments to optimize sessions for
    INDEX_MANAGEMENT_SEGMENTS=1
    # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
    INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
    ```
    * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend
    ```
    # These values are used to handle the Arkime value actions to pivot from Arkime
    #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
    #   them automatically, but they may be specified explicitly here.
    NGINX_DASHBOARDS_PREFIX=
    NGINX_DASHBOARDS_PROXY_PASS=
    ```
    * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
    ```
    # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
    LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
    ```
    ```
    # Zeek log types that will be ignored (dropped) by LogStash
    LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
    ```
    * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation
    ```
    # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
    NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
    NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
    ```
    * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395)
    ```
    # Whether or not enable capture statistics and include them in eve.json
    SURICATA_STATS_ENABLED=false
    SURICATA_STATS_EVE_ENABLED=false
    SURICATA_STATS_INTERVAL=30
    SURICATA_STATS_DECODER_EVENTS=false
    ```
    ```
    # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
    ZEEK_DISABLE_STATS=true
    ```
    * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329)
    ```
    # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
    EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
    ```

.dockerignore

@@ -29,6 +29,7 @@ arkime-raw
 kubernetes
 malcolm-iso
 sensor-iso
+sensor-raspi
 nginx/nginx_ldap*.conf
 pcap
 _site

.github/workflows/api-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/arkime-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/dashboards-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/dashboards-helper-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/dirinit-build-and-push-ghcr.yml

@@ -49,7 +49,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/file-monitor-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/file-upload-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/filebeat-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/freq-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/htadmin-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/logstash-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml

@@ -92,12 +92,12 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Build image
         run: |
-          IMAGES=( $(grep image: docker-compose.yml | awk '{print $2}' | sort -u) )
+          IMAGES=( $(grep image: docker-compose-dev.yml | awk '{print $2}' | sort -u) )
           for IMAGE in "${IMAGES[@]}"; do
             REPO_IMAGE="$(echo "$IMAGE" | sed "s@^.*\(malcolm\)@ghcr.io/${{ github.repository_owner }}/\1@" | sed "s/:.*/:${{ steps.extract_branch.outputs.branch }}/")"
             docker pull "$REPO_IMAGE" && \

.github/workflows/netbox-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/nginx-build-and-push-ghcr.yml

@@ -64,7 +64,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/opensearch-build-and-push-ghcr.yml

@@ -56,7 +56,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/pcap-capture-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/pcap-monitor-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/postgresql-build-and-push-ghcr.yml

@@ -56,7 +56,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/redis-build-and-push-ghcr.yml

@@ -56,7 +56,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/sensor-iso-build-docker-wrap-push-ghcr.yml

@@ -88,7 +88,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Build image

.github/workflows/sensor-raspi-build-docker-wrap-push-ghcr.yml

@@ -0,0 +1,102 @@
+name: sensor-raspi-build-docker-wrap-push-ghcr
+
+on:
+  # push:
+  #   branches:
+  #     - main
+  #     - development
+  #   paths:
+  #     - '.trigger_raspi_workflow_build'
+  workflow_dispatch:
+  # repository_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-22.04
+    permissions:
+      actions: write
+      packages: write
+      contents: read
+      security-events: write
+    defaults:
+      run:
+        shell: bash
+    steps:
+      -
+        name: Cancel previous run in progress
+        uses: styfle/cancel-workflow-action@0.12.0
+        with:
+          ignore_sha: true
+          all_but_latest: true
+          access_token: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+      -
+        name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+        with:
+          driver-opts: |
+            image=moby/buildkit:master
+      -
+        name: Build environment setup
+        run: |
+          sudo apt-get -q update
+          sudo env DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y -q \
+            binfmt-support \
+            bmap-tools \
+            ca-certificates \
+            debootstrap \
+            dosfstools \
+            kpartx \
+            python3 \
+            qemu-user-static \
+            qemu-utils \
+            time \
+            vmdb2 \
+            zerofree
+      -
+        name: Checkout
+        uses: actions/checkout@v4
+      -
+        name: Extract branch name
+        shell: bash
+        run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
+        id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
+      -
+        name: Build image
+        run: |
+          pushd ./sensor-raspi
+          mkdir -p ./shared
+          echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
+          echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
+          echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" > ./shared/environment.chroot
+          echo "VCS_REVSION=${{ steps.extract_commit_sha.outputs.sha }}" > ./shared/environment.chroot
+          echo "BUILD_JOBS=2" > ./shared/environment.chroot
+          sudo make raspi_4_bookworm.img
+          sudo chmod 644 ./raspi_4_bookworm*.*
+          popd
+      -
+        name: ghcr.io login
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Build and push IMG image
+        uses: docker/build-push-action@v5
+        with:
+          context: ./sensor-raspi
+          push: true
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/hedgehog-raspi:${{ steps.extract_branch.outputs.branch }}

.github/workflows/suricata-build-and-push-ghcr.yml

@@ -57,7 +57,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.github/workflows/zeek-build-and-push-ghcr.yml

@@ -56,7 +56,7 @@ jobs:
       -
         name: Extract Malcolm version
         shell: bash
-        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
       -
         name: Set up QEMU

.gitignore

@@ -28,10 +28,12 @@ config.*/
 .envrc
 .direnv
 .vagrant
+.fuse_*
 malcolm_*images.tar.gz
 malcolm_*images.tar.xz
 malcolm_netbox_backup_*.gz
 *.iso
+*.img
 *-build.log
 Gemfile.lock
 _site

.trigger_iso_workflow_build

@@ -1,2 +1,2 @@
 # this file exists solely for the purpose of being updated and seen by github to trigger a commit build action
-2
+3