Skip to content

Parsing error on mdadm reshape #238

@jasonaowen

Description

@jasonaowen

The md_info_detail.sh script generates output that node-exporter fails to parse when a md device is being reshaped.

Error message:

prometheus-node-exporter[4249]: ts=2025-05-08T20:54:34.079Z caller=textfile.go:227 level=error collector=textfile msg="failed to collect textfile data" file=md_info_detail.prom err="failed to parse textfile data from \"/var/lib/prometheus/node-exporter/md_info_detail.prom\": text format parsing error in line 36: expected float as value, got \"2,\""

Line 36 of /var/lib/prometheus/node-exporter/md_info_detail.prom:

node_md_info_DeltaDevices{md_device="md1", md_name="1", raid_level="6", md_num_raid_disks="6 (4)", md_metadata_version="1.2"} 2,

mdadm -D output (partially redacted):

           Version : 1.2
     Creation Time : ...
        Raid Level : raid6
        Array Size : ...
     Used Dev Size : ...
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : ...
             State : clean, reshaping 
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : bitmap

    Reshape Status : ...% complete
     Delta Devices : 2, (4->6)

Version info:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm
$ /sbin/mdadm --version
mdadm - v4.2 - 2021-12-30 - Debian 4.2-5
$ apt policy prometheus-node-exporter prometheus-node-exporter-collectors 
prometheus-node-exporter:
  Installed: 1.5.0-1+b6
  Candidate: 1.5.0-1+b6
prometheus-node-exporter-collectors:
  Installed: 0.0~git20230203.6f710f8-1+deb12u2
  Candidate: 0.0~git20230203.6f710f8-1+deb12u2

I believe this is the command generating that text:

if echo "$line" | grep -E -q "Devices :|Array Size :| Used Dev Size :|Events :"; then
MDADM_DETAIL_KEY=$(echo "$line" | cut -d ":" -f 1 | tr -cd '[a-zA-Z0-9]._-')
MDADM_DETAIL_VALUE=$(echo "$line" | cut -d ":" -f 2 | cut -d " " -f 2 | sed 's:^ ::')
echo "node_md_info_${MDADM_DETAIL_KEY}{md_device=\"${MD_DEVICE_NUM}\", md_name=\"${MD_DEVICE}\", raid_level=\"${MD_LEVEL}\", md_num_raid_disks=\"${MD_NUM_RAID_DISKS}\", md_metadata_version=\"${MD_METADATA_VERSION}\"} ${MDADM_DETAIL_VALUE}"
fi

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions