Jobsub ID 234970.24@dunegpschedd02.fnal.gov

Jobsub ID	234970.24@dunegpschedd02.fnal.gov
Workflow ID	9454
Stage ID	1
User name	mwrobel@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	9437184000 (9000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-29 18:56:46
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-29 20:48:47
From worker node	Hostname	dunegli-7568337-0-fnpc23018.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-29 19:37:25
Input files	usertests:pdhd_prod_beam__231691_30_1_20251008T183131Z_gen_g4_IonScintPDExt.root_227038_141_1_20251010T164547Z_PDInt.root
Jobscript	Exit code	0
	Real time	1h (3877s)
	CPU time	1h (3803s = 98%)
	Max RSS bytes	6677131264 (6367 MiB)
Outputting started	2025-10-29 20:42:03
Output files
Finished	2025-10-29 20:48:47
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

hbours
Iterating with no regularization...
Begin: 4.70333e+07
0 4.29816e+07
1 4.26395e+07
2 4.26015e+07
Now with regularization...
Begin: 4.12714e+07
0 4.12545e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:39:06 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14474!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 89659 traces, 89659 summary
Retagger: tagging trace set: gauss with 71271 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71271 traces tagged "gauss"
FrameSaver: q=2.15171e+06 n=1083293 tag=gauss
wclsFrameSaver: saving 89659 traces tagged "wiener"
FrameSaver: q=2.79383e+06 n=1074253 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12672 XUs and 8643 XVs -> 94 XUVs
C:0 T:1 106007 XUs and 79870 XVs -> 3906 XUVs
C:0 T:2 12778 XUs and 17393 XVs -> 1047 XUVs
C:0 T:3 1 XUs and 5 XVs -> 0 XUVs
C:0 T:5 6890 XUs and 9356 XVs -> 488 XUVs
C:0 T:6 17089 XUs and 33778 XVs -> 2514 XUVs
C:0 T:7 7 XUs and 31 XVs -> 5 XUVs
8054 XUVs total
3116 collection wire objects
8054 potential space points
Neighbour search...
342030 tests to find 127456 neighbours
Iterating with no regularization...
Begin: 4.6687e+07
0 4.25974e+07
1 4.21922e+07
2 4.21503e+07
Now with regularization...
Begin: 3.99828e+07
0 3.99724e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:40:35 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14886!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 88785 traces, 88785 summary
Retagger: tagging trace set: gauss with 71559 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71559 traces tagged "gauss"
FrameSaver: q=2.11416e+06 n=1102408 tag=gauss
wclsFrameSaver: saving 88785 traces tagged "wiener"
FrameSaver: q=2.86489e+06 n=1086428 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13860 XUs and 9096 XVs -> 93 XUVs
C:0 T:1 105174 XUs and 70226 XVs -> 3241 XUVs
C:0 T:2 19515 XUs and 34701 XVs -> 852 XUVs
C:0 T:3 74 XUs and 34 XVs -> 0 XUVs
C:0 T:4 0 XUs and 1 XVs -> 0 XUVs
C:0 T:5 42920 XUs and 26020 XVs -> 895 XUVs
C:0 T:6 16432 XUs and 34413 XVs -> 568 XUVs
C:0 T:7 15 XUs and 34 XVs -> 2 XUVs
5651 XUVs total
2434 collection wire objects
5651 potential space points
Neighbour search...
184151 tests to find 87216 neighbours
Iterating with no regularization...
Begin: 4.28679e+07
0 3.99742e+07
1 3.9776e+07
2 3.97505e+07
Now with regularization...
Begin: 3.82047e+07
0 3.81966e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:42:02 UTC  Closed output file "pdhd_prod_beam__231691_30_1_20251008T183131Z_gen_g4_IonScintPDExt.root_227038_141_1_20251010T164547Z_PDInt.root_234970_24_1_20251029T193728Z_detsim_reco1.root"
29-Oct-2025 20:42:02 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            72.3123       86.9744       103.786       86.608        9.40428        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000394985   0.00377326     0.0072144    0.00360466    0.00226267       10     
reco:opdec:Deconvolution                             0.499753      0.589809       0.65897       0.58914      0.0432875       10     
reco:ophitspe:OpHitFinderDeco                         0.10158      0.108654      0.121764      0.107997     0.00524129       10     
reco:opflash:OpFlashFinder                           0.055434      0.0714475     0.0845277     0.0717161    0.00984084       10     
reco:opslicer:OpSlicer                               0.123783      0.171195      0.220747      0.166858      0.0274612       10     
reco:wclsdatahd:WireCellToolkit                       31.9894       32.3694       32.8588       32.384       0.218972        10     
reco:gaushit:GausHitFinder                           0.367457      0.499252      0.637985      0.479081      0.0781275       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000176928   0.000236515   0.000421686   0.000220436   6.6408e-05       10     
reco:reco3d:SpacePointSolver                         0.395009      0.682659       1.13735       0.62795      0.219633        10     
reco:hitpdune:DisambigFromSpacePoints                0.260537      0.463035      0.769805       0.45464      0.138576        10     
reco:pandora:StandardPandora                          7.8519        15.9877       24.4687       14.8987       4.92367        10     
reco:pandoraWriter:StandardPandora                   0.119274      0.153614      0.234043      0.145744      0.029816        10     
reco:pandoraTrack:LArPandoraTrackCreation            0.636786       1.0273        1.54912       1.01992      0.244537        10     
reco:pandoraShower:LArPandoraShowerCreation          0.479361       1.02839       1.42723       1.02759       0.24751        10     
reco:pandoracalo:Calorimetry                         0.325965      0.564086      0.788053      0.583227      0.127021        10     
reco:pandoracalonosce:Calorimetry                    0.316365      0.553507      0.764758       0.57083      0.124198        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.477409       1.05949       1.32112       1.05525      0.239583        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.465284       1.00532       1.25189      0.996883      0.225997        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0256055     0.0313499     0.0363196     0.0317625    0.00339332       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry       0.02171      0.0296761     0.034604      0.0303841     0.0037164       10     
reco:emtrkmichelid:EmTrackMichelId                    16.7691       20.9086       24.005        20.9776       1.96325        10     
reco:crttag:SingleCRTMatchingProducer               0.00286055    0.00332126    0.00401177    0.00322345    0.000369532      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00206738    0.00269753    0.00301487     0.0027481    0.000276195      10     
reco:pandora2Track:LArPandoraTrackCreation            1.28195       2.57863       3.39258       2.59051      0.530758        10     
reco:pandora2calo:ShowerCalorimetry                  0.422061      0.923166       1.17988      0.899434       0.21742        10     
reco:pandora2calonosce:ShowerCalorimetry             0.414082        0.906        1.16673      0.882537       0.21509        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0232176     0.0315278     0.0377727     0.0322887    0.00402038       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0213986     0.0304078     0.0374511     0.0315834     0.0042393       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.584121       1.15145       1.63713       1.13713       0.26669        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.643177       1.25494       1.68012       1.21999      0.275649        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.632988       1.22318       1.64199       1.18211      0.268839        10     
[art]:TriggerResults:TriggerResultInserter          2.2032e-05    3.50292e-05   0.000122935   2.51185e-05   2.93901e-05      10     
end_path:out1:RootOutput                             3.447e-06    7.6067e-06    3.7632e-05    3.8525e-06    1.01007e-05      10     
end_path:out1:RootOutput(write)                       1.2826        1.50271       1.68704       1.48131      0.132504        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6199.57 MB
  Peak resident set size usage (VmHWM): 3996.82 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11642512
-rw-r--r-- 1 dunegli fnalgrid      39288 Oct 29 20:42 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:42 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:42 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:42 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:42 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:42 time.db
-rw-r--r-- 1 dunegli fnalgrid 6104052715 Oct 29 20:42 pdhd_prod_beam__231691_30_1_20251008T183131Z_gen_g4_IonScintPDExt.root_227038_141_1_20251010T164547Z_PDInt.root_234970_24_1_20251029T193728Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51057620 Oct 29 20:41 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:27 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:26 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:26 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5766514272 Oct 29 20:26 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:37 all-input-dids.txt