Jobsub ID 13371.9@dunegpschedd02.fnal.gov

Jobsub ID	13371.9@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 15:36:00
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2025-07-31 21:24:24
From worker node	Hostname	node2b23.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 16:01:40
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 21:24:24
Output files
Finished	2025-07-31 22:25:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:50 BST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:50 BST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:50 BST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:50 BST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:50 BST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:50 BST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
[2025-07-31 22:24:16.964787 +0100][Debug  ][File              ] [0xe4428b0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ef/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=af0bcf55-d4cc-4df6-8bd8-6de4a83f0d04] Sending a read command for handle 0x0 to pubstor2308.fnal.gov:22242
[2025-07-31 22:24:16.964869 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] MsgHandler created: 0x1a9fcba0 (message: kXR_read (handle: 0x00000000, offset: 2969594211, size: 1368219) ).
[2025-07-31 22:24:16.964981 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] Moving MsgHandler: 0x1a9fcba0 (message: kXR_read (handle: 0x00000000, offset: 2969594211, size: 1368219) ) from out-queu to in-queue.
[2025-07-31 22:24:17.099692 +0100][Debug  ][ExDbgMsg          ] [msg: 0x1ca5f7c0] Assigned MsgHandler: 0x1a9fcba0.
[2025-07-31 22:24:17.099718 +0100][Debug  ][ExDbgMsg          ] [handler: 0x1a9fcba0] Removed MsgHandler: 0x1a9fcba0 from the in-queue.
[2025-07-31 22:24:18.756112 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] Calling MsgHandler: 0x1a9fcba0 (message: kXR_read (handle: 0x00000000, offset: 2969594211, size: 1368219) ) with status: [SUCCESS] .
[2025-07-31 22:24:18.756168 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] Destroying MsgHandler: 0x1a9fcba0.
31-Jul-2025 22:24:20 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T160143Z_reco2.root"
[2025-07-31 22:24:20.094879 +0100][Debug  ][File              ] [0xe4428b0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ef/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=af0bcf55-d4cc-4df6-8bd8-6de4a83f0d04] Sending a close command for handle 0x0 to pubstor2308.fnal.gov:22242
[2025-07-31 22:24:20.095020 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] MsgHandler created: 0xf5105e0 (message: kXR_close (handle: 0x00000000) ).
[2025-07-31 22:24:20.095204 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] Moving MsgHandler: 0xf5105e0 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-31 22:24:20.199218 +0100][Debug  ][ExDbgMsg          ] [msg: 0x30eecb30] Assigned MsgHandler: 0xf5105e0.
[2025-07-31 22:24:20.199265 +0100][Debug  ][ExDbgMsg          ] [handler: 0xf5105e0] Removed MsgHandler: 0xf5105e0 from the in-queue.
[2025-07-31 22:24:20.199431 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] Calling MsgHandler: 0xf5105e0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-31 22:24:20.199549 +0100][Debug  ][File              ] [0xe4428b0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ef/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=af0bcf55-d4cc-4df6-8bd8-6de4a83f0d04] Close returned from pubstor2308.fnal.gov:22242 with: [SUCCESS] 
[2025-07-31 22:24:20.199607 +0100][Debug  ][ExDbgMsg          ] [pubstor2308.fnal.gov:22242] Destroying MsgHandler: 0xf5105e0.
31-Jul-2025 22:24:20 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ef/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1465.79       1905.9        2138.68       1960.06       194.446        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00244825     0.109843      0.914268     0.00763599     0.270871        10     
reco:pandora:StandardPandora                            1308.06       1746.42       1969.8        1801.18       191.813        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000193162   0.00448964     0.0390293    0.000223022    0.0115609       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000221242   0.000929228   0.00500167    0.000247198   0.00143541       10     
reco:pandoracalo:Calorimetry                          0.00028132     0.0017022    0.00268309    0.00190014    0.000684533      10     
reco:pandorapid:Chi2ParticleID                        4.3592e-05    0.000114246   0.000388497   4.6726e-05    0.000128169      10     
reco:linecluster:LineCluster                           0.987137       1.61136       5.43367       1.04818       1.29835        10     
reco:trajcluster:TrajCluster                           0.256349      0.273776      0.288552      0.274198     0.00883531       10     
reco:pmtracktc:PMAlgTrackMaker                          4.84158       5.19496       5.59572       5.19642      0.191976        10     
reco:emtrkmichelid:EmTrackMichelId                      143.362       150.76        160.244       149.393       5.56655        10     
reco:solarflash:SolarOpFlash                            1.15588       1.30181       2.18315       1.17534      0.303026        10     
[art]:TriggerResults:TriggerResultInserter            2.0026e-05    2.5358e-05    4.2281e-05    2.36725e-05   5.88258e-06      10     
end_path:out1:RootOutput                               4.301e-06    5.6456e-06    1.5268e-05     4.616e-06    3.21427e-06      10     
end_path:out1:RootOutput(write)                        0.200714      0.219942      0.307286      0.209514      0.0299674       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3996.05 MB
  Peak resident set size usage (VmHWM): 2525.39 MB
====================================================================================================
[2025-07-31 22:24:20.569695 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-31 22:24:20.570229 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-31 22:24:20.570305 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-31 22:24:20.570565 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-31 22:24:20.570571 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-31 22:24:20.570770 +0100][Debug  ][AsyncSock         ] [pubstor2308.fnal.gov:22242.0] Closing the socket
[2025-07-31 22:24:20.570791 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:52114><--><[::ffff:131.225.69.95]:22242> Removing socket from the poller
[2025-07-31 22:24:20.570835 +0100][Debug  ][PostMaster        ] [pubstor2308.fnal.gov:22242] Destroying stream
[2025-07-31 22:24:20.570841 +0100][Debug  ][AsyncSock         ] [pubstor2308.fnal.gov:22242.0] Closing the socket
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T160143Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T160143Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T160143Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T160143Z_reco2.root.json
reco2_hist.root