Jobsub ID 13237.79@dunegpschedd02.fnal.gov

Jobsub ID	13237.79@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 08:26:39
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2025-07-31 16:36:23
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-07-31 09:57:35
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 16:22:13
Output files
Finished	2025-07-31 16:36:23
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 17:19:15 BST run: 5042 subRun: 0 event: 7770
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 17:19:15 BST run: 5042 subRun: 0 event: 7770
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 17:19:15 BST run: 5042 subRun: 0 event: 7770
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 17:19:15 BST run: 5042 subRun: 0 event: 7770
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 17:19:15 BST run: 5042 subRun: 0 event: 7770
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 17:19:15 BST run: 5042 subRun: 0 event: 7770
Comparing two wires in the same plane: return failure
%MSG
[2025-07-31 17:22:08.607345 +0100][Debug  ][File              ] [0xd936f00@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/aa/c5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=50b9e0a9-3689-4d2d-a301-ba22dd33e23c] Sending a read command for handle 0x0 to stkendca2226.fnal.gov:24474
[2025-07-31 17:22:08.607427 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] MsgHandler created: 0x34c63b30 (message: kXR_read (handle: 0x00000000, offset: 2976146960, size: 1388898) ).
[2025-07-31 17:22:08.607527 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] Moving MsgHandler: 0x34c63b30 (message: kXR_read (handle: 0x00000000, offset: 2976146960, size: 1388898) ) from out-queu to in-queue.
[2025-07-31 17:22:08.734586 +0100][Debug  ][ExDbgMsg          ] [msg: 0x160e9fe0] Assigned MsgHandler: 0x34c63b30.
[2025-07-31 17:22:08.844106 +0100][Debug  ][ExDbgMsg          ] [handler: 0x34c63b30] Removed MsgHandler: 0x34c63b30 from the in-queue.
[2025-07-31 17:22:09.351001 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] Calling MsgHandler: 0x34c63b30 (message: kXR_read (handle: 0x00000000, offset: 2976146960, size: 1388898) ) with status: [SUCCESS] .
[2025-07-31 17:22:09.351061 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] Destroying MsgHandler: 0x34c63b30.
31-Jul-2025 17:22:10 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco_20250731T095738Z_reco2.root"
[2025-07-31 17:22:10.992114 +0100][Debug  ][File              ] [0xd936f00@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/aa/c5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=50b9e0a9-3689-4d2d-a301-ba22dd33e23c] Sending a close command for handle 0x0 to stkendca2226.fnal.gov:24474
[2025-07-31 17:22:10.992379 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] MsgHandler created: 0x308c4f60 (message: kXR_close (handle: 0x00000000) ).
[2025-07-31 17:22:10.992562 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] Moving MsgHandler: 0x308c4f60 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-31 17:22:11.095351 +0100][Debug  ][ExDbgMsg          ] [msg: 0x306fbcc0] Assigned MsgHandler: 0x308c4f60.
[2025-07-31 17:22:11.095401 +0100][Debug  ][ExDbgMsg          ] [handler: 0x308c4f60] Removed MsgHandler: 0x308c4f60 from the in-queue.
[2025-07-31 17:22:11.095537 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] Calling MsgHandler: 0x308c4f60 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-31 17:22:11.095668 +0100][Debug  ][File              ] [0xd936f00@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/aa/c5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=50b9e0a9-3689-4d2d-a301-ba22dd33e23c] Close returned from stkendca2226.fnal.gov:24474 with: [SUCCESS] 
[2025-07-31 17:22:11.095733 +0100][Debug  ][ExDbgMsg          ] [stkendca2226.fnal.gov:24474] Destroying MsgHandler: 0x308c4f60.
31-Jul-2025 17:22:11 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/aa/c5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2080.1        2283.41       2374.44       2310.58       90.0311        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00310501     0.0273863     0.111349      0.0081711     0.0407369       10     
reco:pandora:StandardPandora                            1905.75       2100.73       2187.65       2127.99       87.2805        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000238892   0.00729668     0.0459811    0.00357397     0.0129861       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000249938   0.00144598    0.00455133    0.000873034   0.00124588       10     
reco:pandoracalo:Calorimetry                          0.000179566   0.000716629   0.00293713    0.000277509   0.000839143      10     
reco:pandorapid:Chi2ParticleID                        5.0877e-05    0.000144356   0.000446647   8.50385e-05   0.000136834      10     
reco:linecluster:LineCluster                           0.981854       1.03529       1.43268      0.991761      0.132608        10     
reco:trajcluster:TrajCluster                           0.268222      0.292086      0.329375      0.289527      0.016217        10     
reco:pmtracktc:PMAlgTrackMaker                          4.71548       5.44697       6.02245       5.61074      0.423254        10     
reco:emtrkmichelid:EmTrackMichelId                      166.342       174.439        181.5        174.327       4.57536        10     
reco:solarflash:SolarOpFlash                            1.15778       1.20015       1.38485       1.1665       0.072217        10     
[art]:TriggerResults:TriggerResultInserter            2.6606e-05    3.05428e-05   4.7444e-05    2.9269e-05    5.74544e-06      10     
end_path:out1:RootOutput                               4.601e-06    6.3749e-06    1.6815e-05    5.2745e-06    3.48812e-06      10     
end_path:out1:RootOutput(write)                        0.208367      0.232854      0.348005      0.217029      0.0406009       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4007.21 MB
  Peak resident set size usage (VmHWM): 2535.86 MB
====================================================================================================
[2025-07-31 17:22:11.559910 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-31 17:22:11.560541 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-31 17:22:11.560622 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-31 17:22:11.560757 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-31 17:22:11.560764 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-31 17:22:11.560968 +0100][Debug  ][AsyncSock         ] [stkendca2226.fnal.gov:24474.0] Closing the socket
[2025-07-31 17:22:11.560980 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:55792><--><[::ffff:131.225.69.191]:24474> Removing socket from the poller
[2025-07-31 17:22:11.561013 +0100][Debug  ][PostMaster        ] [stkendca2226.fnal.gov:24474] Destroying stream
[2025-07-31 17:22:11.561019 +0100][Debug  ][AsyncSock         ] [stkendca2226.fnal.gov:24474.0] Closing the socket
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco_20250731T095738Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco_20250731T095738Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco_20250731T095738Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T064741Z_gen_000777_supernova_g4stage1_g4stage2_detsim_reco_20250731T095738Z_reco2.root.json
reco2_hist.root