I am trying to parse the output of a command from the EMC OneFS CLI. We are trying to collect inventory data, and since we're on an older version of the API we can't use the REST SDK.
Instead we have decided to do it the "hacky" way and parse the output of CLI commands.
How can I reliably parse this data in Python, given the varying formats of text? Notice the varying amounts of white space, and the fact that some of these are separated by :
while others are separated by =
.
I have a decent start on this, I was able to split up each line into a list and handle the cases where the =
sign is nested within parenthesis.
I am still stuck on the other parts:
- Take the first column as the "hostname"
- Build a separate dictionary for each hostname that contains all key value pairs (some are separated by
:
others are separated by=
)
RAW OUTPUT THAT NEEDS TO BE PARSED:
visi-91: SerNo: S123456789P
visi-91: Config: 912-111-231
visi-91: Class: storage
visi-91: Series: x_series
visi-91: Product: IQ 32000x-ssd
visi-91: Comps: Chassis CPU Motherboard NVRam FlashDrive Disk_Controller Disk_Expander Power_Supplies
visi-91: HWGen: MaunaLoa (MaunaLoa Hardware)
visi-91: Chassis: ISI36 (Isilon (CIDesigns) 36-Bay Chassis)
visi-91: CPU: GenuineIntel (2.33GHz, stepping 0x0001067a)
visi-91: PROC: Dual-proc, Quad-core
visi-91: RAM: 17150812160 Bytes
visi-91: Mobo: SupMicX7DBU (SuperMicro X7DBU Motherboard)
visi-91: NVRam: MT25208 (Mellanox 25208 based Card) (492MB card) (size 515899392B)
visi-91: FlshDrv: None (No physical dongle supported) ((null))
visi-91: DskCtl: LSI3081E-HBA (LSI SAS3081E SAS Controller -- HBA) (8 ports)
visi-91: DskExp: VIT7156 (Vitesse 7156 Disk Expander)
visi-91: PwrSupl: PS1 (type=Emerson, fw=v.01.00.00)
visi-91: PwrSupl: PS2 (type=Emerson, fw=v.01.00.00)
visi-91: ChasCnt: 1 (Single-Chassis System)
visi-91: NetIF: em*4,ib*2 (4x Intel, 2x IB)
visi-91: IBType: ISI25208 (Isilon MT25208-based IB Card)
visi-91: LCDver: NoriVFD (Noritake VFD)
visi-91: IMB: Board Version 0x20f
visi-91: Power Supplies OK
visi-91: CPU Operation (raw 0x882F0000) = Normal
visi-91: CPU Speed Limit = 100.00%
visi-91: Chassis Fan1 (ISI F1) = 7469
visi-91: Chassis Fan2 (ISI F2) = 7552
visi-91: Chassis Fan3 (ISI F3) = 7387
visi-91: Power Supply 1 Fan = 5504
visi-91: Power Supply 2 Fan = 5792
visi-91: Temp Until CPU Throttle (CPU 0) = 46.0
visi-91: Temp Until CPU Throttle (CPU 1) = 42.0
visi-91: Temp Front Panel = 17.0
visi-91: Temp Chassis 2 (ISI T2) = 22.0
visi-91: Temp System = 25.0
visi-92: SerNo: S123456785K
visi-92: Config: 912-111-231
visi-92: Class: storage
visi-92: Series: x_series
visi-92: Product: IQ 32000x-ssd
visi-92: Comps: Chassis CPU Motherboard NVRam FlashDrive Disk_Controller Disk_Expander Power_Supplies
visi-92: HWGen: MaunaLoa (MaunaLoa Hardware)
visi-92: Chassis: ISI36 (Isilon (CIDesigns) 36-Bay Chassis)
visi-92: CPU: GenuineIntel (2.33GHz, stepping 0x0001067a)
visi-92: PROC: Dual-proc, Quad-core
visi-92: RAM: 17150812160 Bytes
visi-92: Mobo: SupMicX7DBU (SuperMicro X7DBU Motherboard)
visi-92: NVRam: MT25208 (Mellanox 25208 based Card) (492MB card) (size 515899392B)
visi-92: FlshDrv: None (No physical dongle supported) ((null))
visi-92: DskCtl: LSI3081E-HBA (LSI SAS3081E SAS Controller -- HBA) (8 ports)
visi-92: DskExp: VIT7156 (Vitesse 7156 Disk Expander)
visi-92: PwrSupl: PS1 (type=Emerson, fw=v.01.00.00)
visi-92: PwrSupl: PS2 (type=Emerson, fw=v.01.00.00)
visi-92: ChasCnt: 1 (Single-Chassis System)
visi-92: NetIF: em*4,ib*2 (4x Intel, 2x IB)
visi-92: IBType: ISI25208 (Isilon MT25208-based IB Card)
visi-92: LCDver: NoriVFD (Noritake VFD)
visi-92: IMB: Board Version 0x20f
visi-92: Power Supplies OK
visi-92: CPU Operation (raw 0x882E0000) = Normal
visi-92: CPU Speed Limit = 100.00%
visi-92: Chassis Fan1 (ISI F1) = 7268
visi-92: Chassis Fan2 (ISI F2) = 7307
visi-92: Chassis Fan3 (ISI F3) = 7190
visi-92: Power Supply 1 Fan = 5376
visi-92: Power Supply 2 Fan = 5696
visi-92: Temp Until CPU Throttle (CPU 0) = 46.0
visi-92: Temp Until CPU Throttle (CPU 1) = 46.0
visi-92: Temp Front Panel = 17.2
visi-92: Temp Chassis 2 (ISI T2) = 24.0
visi-92: Temp System = 27.0
visi-93: SerNo: S987654321P
visi-93: Config: 912-111-231
visi-93: Class: storage
visi-93: Series: x_series
visi-93: Product: IQ 32000x-ssd
visi-93: Comps: Chassis CPU Motherboard NVRam FlashDrive Disk_Controller Disk_Expander Power_Supplies
visi-93: HWGen: MaunaLoa (MaunaLoa Hardware)
visi-93: Chassis: ISI36 (Isilon (CIDesigns) 36-Bay Chassis)
visi-93: CPU: GenuineIntel (2.33GHz, stepping 0x0001067a)
visi-93: PROC: Dual-proc, Quad-core
visi-93: RAM: 17150812160 Bytes
visi-93: Mobo: SupMicX7DBU (SuperMicro X7DBU Motherboard)
visi-93: NVRam: MT25208 (Mellanox 25208 based Card) (492MB card) (size 515899392B)
visi-93: FlshDrv: None (No physical dongle supported) ((null))
visi-93: DskCtl: LSI3081E-HBA (LSI SAS3081E SAS Controller -- HBA) (8 ports)
visi-93: DskExp: VIT7156 (Vitesse 7156 Disk Expander)
visi-93: PwrSupl: PS1 (type=Emerson, fw=v.03.05.00)
visi-93: PwrSupl: PS2 (type=Emerson, fw=v.03.05.00)
visi-93: ChasCnt: 1 (Single-Chassis System)
visi-93: NetIF: em*4,ib*2 (4x Intel, 2x IB)
visi-93: IBType: ISI25208 (Isilon MT25208-based IB Card)
visi-93: LCDver: NoriVFD (Noritake VFD)
visi-93: IMB: Board Version 0x20f
visi-93: Power Supplies OK
visi-93: CPU Operation (raw 0x88250000) = Normal
visi-93: CPU Speed Limit = 100.00%
visi-93: Chassis Fan1 (ISI F1) = 7387
visi-93: Chassis Fan2 (ISI F2) = 7152
visi-93: Chassis Fan3 (ISI F3) = 7152
visi-93: Power Supply 1 Fan = 8512
visi-93: Power Supply 2 Fan = 8512
visi-93: Temp Until CPU Throttle (CPU 0) = 37.0
visi-93: Temp Until CPU Throttle (CPU 1) = 38.0
visi-93: Temp Front Panel = 17.4
visi-93: Temp Chassis 2 (ISI T2) = 25.0
visi-93: Temp System = 28.0
MY CODE SO FAR:
#Read text file
file = open("test_itry.txt", "r")
lines = file.readlines()
file.close()
for line in lines:
#Remove whitespaces
line = line.replace(' ','')
#Remove newlines
line = line.replace('\n','')
#Handler values that have equals signs nested within parenthesis
#If line has closed parenthesis and equals sign
if line.count(")") > 0 and line.count("=") > 0:
#Get first position of open parenthesis
poscp = line.find(")")
#Get first position of equals sign
poseq = line.find("=")
#If position of equals sign comes before position of first parenthesis
if poscp < poseq:
#Split by first equals sign
line = line.split("=", 1)
#If line doesn't have closed parenthesis and an equals sign
else:
line = line.split("=", 1)
#If line has not been split into a list yet create a list
if isinstance(line, str):
line = line.split()
print(line)
OUTPUT OF MY CODE SO FAR:
['visi-91:SerNo:S123456789P']
['visi-91:Config:912-111-231']
['visi-91:Class:storage']
['visi-91:Series:x_series']
['visi-91:Product:IQ32000x-ssd']
['visi-91:Comps:ChassisCPUMotherboardNVRamFlashDriveDisk_ControllerDisk_ExpanderPower_Supplies']
['visi-91:HWGen:MaunaLoa(MaunaLoaHardware)']
['visi-91:Chassis:ISI36(Isilon(CIDesigns)36-BayChassis)']
['visi-91:CPU:GenuineIntel(2.33GHz,stepping0x0001067a)']
['visi-91:PROC:Dual-proc,Quad-core']
['visi-91:RAM:17150812160Bytes']
['visi-91:Mobo:SupMicX7DBU(SuperMicroX7DBUMotherboard)']
['visi-91:NVRam:MT25208(Mellanox25208basedCard)(492MBcard)(size515899392B)']
['visi-91:FlshDrv:None(Nophysicaldonglesupported)((null))']
['visi-91:DskCtl:LSI3081E-HBA(LSISAS3081ESASController--HBA)(8ports)']
['visi-91:DskExp:VIT7156(Vitesse7156DiskExpander)']
['visi-91:PwrSupl:PS1(type=Emerson,fw=v.01.00.00)']
['visi-91:PwrSupl:PS2(type=Emerson,fw=v.01.00.00)']
['visi-91:ChasCnt:1(Single-ChassisSystem)']
['visi-91:NetIF:em*4,ib*2(4xIntel,2xIB)']
['visi-91:IBType:ISI25208(IsilonMT25208-basedIBCard)']
['visi-91:LCDver:NoriVFD(NoritakeVFD)']
['visi-91:IMB:BoardVersion0x20f']
['visi-91:PowerSuppliesOK']
['visi-91:CPUOperation(raw0x882F0000)', 'Normal']
['visi-91:CPUSpeedLimit', '100.00%']
['visi-91:ChassisFan1(ISIF1)', '7469']
['visi-91:ChassisFan2(ISIF2)', '7552']
['visi-91:ChassisFan3(ISIF3)', '7387']
['visi-91:PowerSupply1Fan', '5504']
['visi-91:PowerSupply2Fan', '5792']
['visi-91:TempUntilCPUThrottle(CPU0)', '46.0']
['visi-91:TempUntilCPUThrottle(CPU1)', '42.0']
['visi-91:TempFrontPanel', '17.0']
['visi-91:TempChassis2(ISIT2)', '22.0']
['visi-91:TempSystem', '25.0']
['visi-92:SerNo:S123456785K']
['visi-92:Config:912-111-231']
['visi-92:Class:storage']
['visi-92:Series:x_series']
['visi-92:Product:IQ32000x-ssd']
['visi-92:Comps:ChassisCPUMotherboardNVRamFlashDriveDisk_ControllerDisk_ExpanderPower_Supplies']
['visi-92:HWGen:MaunaLoa(MaunaLoaHardware)']
['visi-92:Chassis:ISI36(Isilon(CIDesigns)36-BayChassis)']
['visi-92:CPU:GenuineIntel(2.33GHz,stepping0x0001067a)']
['visi-92:PROC:Dual-proc,Quad-core']
['visi-92:RAM:17150812160Bytes']
['visi-92:Mobo:SupMicX7DBU(SuperMicroX7DBUMotherboard)']
['visi-92:NVRam:MT25208(Mellanox25208basedCard)(492MBcard)(size515899392B)']
['visi-92:FlshDrv:None(Nophysicaldonglesupported)((null))']
['visi-92:DskCtl:LSI3081E-HBA(LSISAS3081ESASController--HBA)(8ports)']
['visi-92:DskExp:VIT7156(Vitesse7156DiskExpander)']
['visi-92:PwrSupl:PS1(type=Emerson,fw=v.01.00.00)']
['visi-92:PwrSupl:PS2(type=Emerson,fw=v.01.00.00)']
['visi-92:ChasCnt:1(Single-ChassisSystem)']
['visi-92:NetIF:em*4,ib*2(4xIntel,2xIB)']
['visi-92:IBType:ISI25208(IsilonMT25208-basedIBCard)']
['visi-92:LCDver:NoriVFD(NoritakeVFD)']
['visi-92:IMB:BoardVersion0x20f']
['visi-92:PowerSuppliesOK']
['visi-92:CPUOperation(raw0x882E0000)', 'Normal']
['visi-92:CPUSpeedLimit', '100.00%']
['visi-92:ChassisFan1(ISIF1)', '7268']
['visi-92:ChassisFan2(ISIF2)', '7307']
['visi-92:ChassisFan3(ISIF3)', '7190']
['visi-92:PowerSupply1Fan', '5376']
['visi-92:PowerSupply2Fan', '5696']
['visi-92:TempUntilCPUThrottle(CPU0)', '46.0']
['visi-92:TempUntilCPUThrottle(CPU1)', '46.0']
['visi-92:TempFrontPanel', '17.2']
['visi-92:TempChassis2(ISIT2)', '24.0']
['visi-92:TempSystem', '27.0']
['visi-93:SerNo:S987654321P']
['visi-93:Config:912-111-231']
['visi-93:Class:storage']
['visi-93:Series:x_series']
['visi-93:Product:IQ32000x-ssd']
['visi-93:Comps:ChassisCPUMotherboardNVRamFlashDriveDisk_ControllerDisk_ExpanderPower_Supplies']
['visi-93:HWGen:MaunaLoa(MaunaLoaHardware)']
['visi-93:Chassis:ISI36(Isilon(CIDesigns)36-BayChassis)']
['visi-93:CPU:GenuineIntel(2.33GHz,stepping0x0001067a)']
['visi-93:PROC:Dual-proc,Quad-core']
['visi-93:RAM:17150812160Bytes']
['visi-93:Mobo:SupMicX7DBU(SuperMicroX7DBUMotherboard)']
['visi-93:NVRam:MT25208(Mellanox25208basedCard)(492MBcard)(size515899392B)']
['visi-93:FlshDrv:None(Nophysicaldonglesupported)((null))']
['visi-93:DskCtl:LSI3081E-HBA(LSISAS3081ESASController--HBA)(8ports)']
['visi-93:DskExp:VIT7156(Vitesse7156DiskExpander)']
['visi-93:PwrSupl:PS1(type=Emerson,fw=v.03.05.00)']
['visi-93:PwrSupl:PS2(type=Emerson,fw=v.03.05.00)']
['visi-93:ChasCnt:1(Single-ChassisSystem)']
['visi-93:NetIF:em*4,ib*2(4xIntel,2xIB)']
['visi-93:IBType:ISI25208(IsilonMT25208-basedIBCard)']
['visi-93:LCDver:NoriVFD(NoritakeVFD)']
['visi-93:IMB:BoardVersion0x20f']
['visi-93:PowerSuppliesOK']
['visi-93:CPUOperation(raw0x88250000)', 'Normal']
['visi-93:CPUSpeedLimit', '100.00%']
['visi-93:ChassisFan1(ISIF1)', '7387']
['visi-93:ChassisFan2(ISIF2)', '7152']
['visi-93:ChassisFan3(ISIF3)', '7152']
['visi-93:PowerSupply1Fan', '8512']
['visi-93:PowerSupply2Fan', '8512']
['visi-93:TempUntilCPUThrottle(CPU0)', '37.0']
['visi-93:TempUntilCPUThrottle(CPU1)', '38.0']
['visi-93:TempFrontPanel', '17.4']
['visi-93:TempChassis2(ISIT2)', '25.0']
['visi-93:TempSystem', '28.0']
DESIRED OUTPUT - Dictionary key/value pairs with first word as hostname. Separate dictionaries for each hostname:
{'hostname':'visi-91', 'SerNo':'S123456789P', 'Config':'912-111-231', 'Class':'storage', 'Series':'x_series', 'PwrSupl':'PS1(type=Emerson,fw=v.01.00.00)', 'Chassis':'ISI36(Isilon(CIDesigns)36-BayChassis)', 'NetIF':'em*4,ib*2(4xIntel,2xIB)', 'IBType':'ISI25208(IsilonMT25208-basedIBCard)'} //etc...
{'hostname':'visi-92', 'SerNo':'S123456785K', 'Config':'912-111-231', 'Class':'storage', 'Series':'x_series', 'PwrSupl':'PS1(type=Emerson,fw=v.01.00.00)' 'Chassis':'ISI36(Isilon(CIDesigns)36-BayChassis)', 'NetIF':'em*4,ib*2(4xIntel,2xIB)', 'IBType':'ISI25208(IsilonMT25208-basedIBCard)'} //etc...