justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.12@dunegpschedd02.fnal.gov

Jobsub ID234970.12@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:50:18
From worker nodeHostnamedunegli-7568267-0-fnpc23023.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:38:09
Input filesusertests:pdhd_prod_beam__231691_59_1_20251008T183247Z_gen_g4_IonScintPDExt.root_227038_124_1_20251010T163910Z_PDInt.root
JobscriptExit code0
Real time1h (3923s)
CPU time1h (3794s = 96%)
Max RSS bytes7107108864 (6777 MiB)
Outputting started2025-10-29 20:43:32
Output files
Finished2025-10-29 20:50:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ation...
Begin: 1.50563e+07
0 1.39357e+07
1 1.38555e+07
2 1.38405e+07
3 1.38361e+07
Now with regularization...
Begin: 1.26963e+07
0 1.26939e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:40:16 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16730!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 95150 traces, 95150 summary
Retagger: tagging trace set: gauss with 77936 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77936 traces tagged "gauss"
FrameSaver: q=2.42889e+06 n=1255950 tag=gauss
wclsFrameSaver: saving 95150 traces tagged "wiener"
FrameSaver: q=3.03023e+06 n=1206342 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14814 XUs and 11432 XVs -> 108 XUVs
C:0 T:1 79646 XUs and 59567 XVs -> 2161 XUVs
C:0 T:2 36475 XUs and 20686 XVs -> 1074 XUVs
C:0 T:3 5 XUs and 10 XVs -> 0 XUVs
C:0 T:4 3 XUs and 4 XVs -> 2 XUVs
C:0 T:5 25018 XUs and 21500 XVs -> 731 XUVs
C:0 T:6 13662 XUs and 7795 XVs -> 243 XUVs
C:0 T:7 0 XUs and 4 XVs -> 0 XUVs
4319 XUVs total
2599 collection wire objects
4319 potential space points
Neighbour search...
141037 tests to find 52396 neighbours
Iterating with no regularization...
Begin: 4.16684e+07
0 3.87784e+07
1 3.86208e+07
2 3.86154e+07
Now with regularization...
Begin: 3.69165e+07
0 3.69138e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:41:49 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17930!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 107638 traces, 107638 summary
Retagger: tagging trace set: gauss with 84633 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84633 traces tagged "gauss"
FrameSaver: q=2.62835e+06 n=1361748 tag=gauss
wclsFrameSaver: saving 107638 traces tagged "wiener"
FrameSaver: q=3.2938e+06 n=1289913 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9972 XUs and 7500 XVs -> 56 XUVs
C:0 T:1 78803 XUs and 46855 XVs -> 2305 XUVs
C:0 T:2 53588 XUs and 124205 XVs -> 3981 XUVs
C:0 T:3 39 XUs and 30 XVs -> 0 XUVs
C:0 T:5 54789 XUs and 31414 XVs -> 1051 XUVs
C:0 T:6 23924 XUs and 32602 XVs -> 411 XUVs
C:0 T:7 14 XUs and 14 XVs -> 3 XUVs
7807 XUVs total
3198 collection wire objects
7807 potential space points
Neighbour search...
551539 tests to find 236704 neighbours
Iterating with no regularization...
Begin: 5.02567e+07
0 4.33618e+07
1 4.29371e+07
2 4.28883e+07
3 4.28732e+07
Now with regularization...
Begin: 4.11672e+07
0 4.11553e+07
29-Oct-2025 20:43:31 UTC  Closed output file "pdhd_prod_beam__231691_59_1_20251008T183247Z_gen_g4_IonScintPDExt.root_227038_124_1_20251010T163910Z_PDInt.root_234970_12_1_20251029T193811Z_detsim_reco1.root"
29-Oct-2025 20:43:32 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            68.4191       89.0761       108.078       89.0594       11.2045        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000396369   0.00402311    0.00703748    0.00425467    0.00216915       10     
reco:opdec:Deconvolution                             0.525229      0.603403      0.672817      0.605188      0.0411377       10     
reco:ophitspe:OpHitFinderDeco                        0.100553      0.104274      0.107168      0.104164     0.00230791       10     
reco:opflash:OpFlashFinder                           0.0554819     0.073053      0.0899137     0.070428      0.0115915       10     
reco:opslicer:OpSlicer                               0.142078      0.180638       0.21989      0.178487      0.0207537       10     
reco:wclsdatahd:WireCellToolkit                       31.2714       31.6803       32.4718       31.6037      0.381227        10     
reco:gaushit:GausHitFinder                           0.428671      0.526304      0.635337      0.533313      0.0687922       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000194703   0.000274789   0.000493454   0.000250951   8.82741e-05      10     
reco:reco3d:SpacePointSolver                         0.236157      0.520681      0.727897      0.611825      0.164845        10     
reco:hitpdune:DisambigFromSpacePoints                0.257686      0.443893      0.599041      0.449234      0.112295        10     
reco:pandora:StandardPandora                          7.40528       17.1591       34.8095       14.8214       7.50826        10     
reco:pandoraWriter:StandardPandora                   0.138979      0.165185      0.220805      0.163887      0.0228839       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.777468       1.07986       1.59748      0.977022      0.278784        10     
reco:pandoraShower:LArPandoraShowerCreation          0.663559       1.07518       1.79399       1.07197      0.333895        10     
reco:pandoracalo:Calorimetry                         0.431672      0.613332      0.877718       0.55366      0.158608        10     
reco:pandoracalonosce:Calorimetry                    0.424523      0.602494      0.865306      0.541133      0.156722        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.683541       1.14793       2.10872       1.14376      0.397208        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.666602       1.11646       2.05806       1.09955      0.383207        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0246439     0.033917      0.0411795     0.0345379    0.00578279       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0241406     0.032018      0.0409076     0.0318644    0.00532542       10     
reco:emtrkmichelid:EmTrackMichelId                     17.61        21.5482       25.2798       21.4296       2.55676        10     
reco:crttag:SingleCRTMatchingProducer               0.00261649    0.00382139    0.00511842    0.00388748    0.000647594      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00228518    0.00290155    0.00361322    0.00294957    0.000364238      10     
reco:pandora2Track:LArPandoraTrackCreation            1.66494       2.77991       4.18655       2.63091      0.848211        10     
reco:pandora2calo:ShowerCalorimetry                  0.558668       1.01572       1.81001       1.00776      0.351881        10     
reco:pandora2calonosce:ShowerCalorimetry             0.544427      0.991276       1.76785      0.985563      0.345667        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0258442     0.0334245     0.0418247     0.0326215    0.00549946       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0248486     0.0322398     0.0397607     0.0315284    0.00557218       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.699898       1.21875       1.82257       1.18032      0.356446        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.787134       1.37546        2.245        1.32784       0.43887        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.765166       1.34326       2.19168       1.29963      0.426808        10     
[art]:TriggerResults:TriggerResultInserter           2.064e-05    3.61151e-05   0.000120791   2.72775e-05   2.85661e-05      10     
end_path:out1:RootOutput                             5.681e-06    1.19709e-05   3.4216e-05     9.939e-06    7.57509e-06      10     
end_path:out1:RootOutput(write)                       1.24437       1.50835       1.74098       1.52685      0.151548        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6189.43 MB
  Peak resident set size usage (VmHWM): 3993.01 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12137836
-rw-r--r-- 1 dunegli fnalgrid      39001 Oct 29 20:43 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:43 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:43 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:43 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:43 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:43 time.db
-rw-r--r-- 1 dunegli fnalgrid 6366323912 Oct 29 20:43 pdhd_prod_beam__231691_59_1_20251008T183247Z_gen_g4_IonScintPDExt.root_227038_124_1_20251010T163910Z_PDInt.root_234970_12_1_20251029T193811Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   53753280 Oct 29 20:42 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:28 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:27 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:26 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6008751321 Oct 29 20:26 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:38 all-input-dids.txt
justIN time: 2025-11-03 18:32:48 UTC       justIN version: 01.05.01