Jobsub ID 234970.7@dunegpschedd02.fnal.gov

Jobsub ID	234970.7@dunegpschedd02.fnal.gov
Workflow ID	9454
Stage ID	1
User name	mwrobel@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	9437184000 (9000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-29 18:56:46
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-29 20:48:38
From worker node	Hostname	dunegli-7568311-0-fnpc23022.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-29 19:37:56
Input files	usertests:pdhd_prod_beam__231691_47_1_20251008T183226Z_gen_g4_IonScintPDExt.root_227038_29_1_20251010T161224Z_PDInt.root
Jobscript	Exit code	0
	Real time	1h (3838s)
	CPU time	1h (3752s = 97%)
	Max RSS bytes	6877757440 (6559 MiB)
Outputting started	2025-10-29 20:41:55
Output files
Finished	2025-10-29 20:48:38
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

2971 collection wire objects
7385 potential space points
Neighbour search...
514379 tests to find 165548 neighbours
Iterating with no regularization...
Begin: 5.39835e+07
0 4.76995e+07
1 4.71381e+07
2 4.71e+07
Now with regularization...
Begin: 4.54789e+07
0 4.54676e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:38:31 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14070!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 82729 traces, 82729 summary
Retagger: tagging trace set: gauss with 68044 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68044 traces tagged "gauss"
FrameSaver: q=2.05411e+06 n=1109968 tag=gauss
wclsFrameSaver: saving 82729 traces tagged "wiener"
FrameSaver: q=2.51504e+06 n=1082674 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8386 XUs and 7439 XVs -> 49 XUVs
C:0 T:1 39268 XUs and 32428 XVs -> 805 XUVs
C:0 T:2 5897 XUs and 8199 XVs -> 936 XUVs
C:0 T:4 15 XUs and 24 XVs -> 2 XUVs
C:0 T:5 12956 XUs and 18109 XVs -> 1481 XUVs
C:0 T:6 22318 XUs and 19006 XVs -> 652 XUVs
3925 XUVs total
2098 collection wire objects
3925 potential space points
Neighbour search...
70619 tests to find 40094 neighbours
Iterating with no regularization...
Begin: 2.4951e+07
0 2.35755e+07
1 2.34596e+07
2 2.3447e+07
Now with regularization...
Begin: 2.16422e+07
0 2.16391e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:39:50 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17635!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 117793 traces, 117793 summary
Retagger: tagging trace set: gauss with 92482 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 92482 traces tagged "gauss"
FrameSaver: q=3.09259e+06 n=1585211 tag=gauss
wclsFrameSaver: saving 117793 traces tagged "wiener"
FrameSaver: q=3.82411e+06 n=1520066 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11332 XUs and 10302 XVs -> 81 XUVs
C:0 T:1 79762 XUs and 58464 XVs -> 1998 XUVs
C:0 T:2 49059 XUs and 42911 XVs -> 1126 XUVs
C:0 T:5 45904 XUs and 38107 XVs -> 1311 XUVs
C:0 T:6 30980 XUs and 56085 XVs -> 1056 XUVs
C:0 T:7 10 XUs and 17 XVs -> 1 XUVs
5573 XUVs total
3564 collection wire objects
5573 potential space points
Neighbour search...
127381 tests to find 60480 neighbours
Iterating with no regularization...
Begin: 6.74032e+07
0 6.39954e+07
1 6.38188e+07
2 6.38073e+07
Now with regularization...
Begin: 6.17015e+07
0 6.16976e+07
29-Oct-2025 20:41:54 UTC  Closed output file "pdhd_prod_beam__231691_47_1_20251008T183226Z_gen_g4_IonScintPDExt.root_227038_29_1_20251010T161224Z_PDInt.root_234970_7_1_20251029T193758Z_detsim_reco1.root"
29-Oct-2025 20:41:54 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            68.6532       88.3316       121.218       83.7213       15.4606        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00046153    0.00376744    0.00620554    0.00391279    0.00203104       10     
reco:opdec:Deconvolution                              0.50379      0.598612      0.734065      0.586839      0.0584787       10     
reco:ophitspe:OpHitFinderDeco                         0.10446      0.107037      0.113066      0.106031     0.00270575       10     
reco:opflash:OpFlashFinder                           0.0458436     0.0674821     0.0925951     0.068214      0.0145699       10     
reco:opslicer:OpSlicer                               0.128225      0.165145      0.207082      0.159939      0.0201426       10     
reco:wclsdatahd:WireCellToolkit                       32.5699       32.881        33.3384       32.9237       0.2719         10     
reco:gaushit:GausHitFinder                           0.382466      0.526154      0.800253      0.504596      0.109615        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000202866   0.000254192   0.000356169   0.00023377    4.70831e-05      10     
reco:reco3d:SpacePointSolver                          0.23974      0.660505       1.59174      0.518284      0.437121        10     
reco:hitpdune:DisambigFromSpacePoints                0.242376      0.427948      0.665283      0.367295      0.144745        10     
reco:pandora:StandardPandora                          9.76126       15.0162       24.4455       13.7256       4.42844        10     
reco:pandoraWriter:StandardPandora                   0.120663      0.165192      0.222206      0.165802      0.0324262       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.450039      0.992599       1.63831      0.884783      0.316997        10     
reco:pandoraShower:LArPandoraShowerCreation          0.447903       1.07231       2.20531      0.925693      0.543352        10     
reco:pandoracalo:Calorimetry                          0.25713      0.562482      0.950911      0.503136      0.184421        10     
reco:pandoracalonosce:Calorimetry                    0.248345       0.55439      0.936082      0.489342      0.184366        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.465192       1.16143       2.58193      0.982857      0.652459        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.457225       1.0953        2.41032      0.930114      0.605316        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0252219     0.0328524     0.0461979     0.0320776    0.00607455       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0216977     0.0314456     0.0437722     0.0313132    0.00573678       10     
reco:emtrkmichelid:EmTrackMichelId                    16.6561       21.8973       29.7038       21.1838       3.86398        10     
reco:crttag:SingleCRTMatchingProducer               0.00250675    0.00381271    0.00541729    0.00372283    0.000868765      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00219873    0.00297133    0.00417939    0.00300812    0.000525056      10     
reco:pandora2Track:LArPandoraTrackCreation            1.30229       2.69401       5.04422       2.32867       1.16905        10     
reco:pandora2calo:ShowerCalorimetry                  0.448739       1.03529       2.28723       0.85751      0.584247        10     
reco:pandora2calonosce:ShowerCalorimetry             0.444428       1.01376       2.26272      0.841991       0.56498        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0235984     0.0331021     0.0463393     0.0335559    0.00628587       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0232032     0.0318796     0.0439198     0.0320057    0.00568309       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.567908       1.20067       2.2861        1.0647       0.515414        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.617217       1.36378       2.83663       1.15006      0.665718        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.604509       1.32645       2.78317       1.1295       0.651269        10     
[art]:TriggerResults:TriggerResultInserter          1.8124e-05    2.9926e-05    9.7886e-05    2.2603e-05    2.28507e-05      10     
end_path:out1:RootOutput                             2.865e-06    6.7119e-06    2.4317e-05     5.09e-06     6.08819e-06      10     
end_path:out1:RootOutput(write)                       1.23931       1.54126       2.06376       1.45227      0.253333        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6143.14 MB
  Peak resident set size usage (VmHWM): 3944.86 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12276480
-rw-r--r-- 1 dunegli fnalgrid      39193 Oct 29 20:41 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Oct 29 20:41 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:41 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:41 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:41 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:41 time.db
-rw-r--r-- 1 dunegli fnalgrid 6430517215 Oct 29 20:41 pdhd_prod_beam__231691_47_1_20251008T183226Z_gen_g4_IonScintPDExt.root_227038_29_1_20251010T161224Z_PDInt.root_234970_7_1_20251029T193758Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   52948080 Oct 29 20:40 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:27 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:25 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:25 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6087338541 Oct 29 20:25 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Oct 29 19:37 all-input-dids.txt