justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.15@dunegpschedd02.fnal.gov

Jobsub ID234970.15@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:48:34
From worker nodeHostnamedunegli-7568339-0-fnpc23022.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:34
Input filesusertests:pdhd_prod_beam__231691_35_1_20251008T183122Z_gen_g4_IonScintPDExt.root_227038_36_1_20251010T161213Z_PDInt.root
JobscriptExit code0
Real time1h (3857s)
CPU time1h (3780s = 98%)
Max RSS bytes7216345088 (6882 MiB)
Outputting started2025-10-29 20:41:51
Output files
Finished2025-10-29 20:48:34
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ts to find 466362 neighbours
Iterating with no regularization...
Begin: 3.39047e+08
0 2.79968e+08
1 2.74889e+08
2 2.74514e+08
3 2.74459e+08
Now with regularization...
Begin: 2.47085e+08
0 2.46696e+08
1 2.46635e+08
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:38:49 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13318!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 84681 traces, 84681 summary
Retagger: tagging trace set: gauss with 68435 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68435 traces tagged "gauss"
FrameSaver: q=2.07738e+06 n=1023343 tag=gauss
wclsFrameSaver: saving 84681 traces tagged "wiener"
FrameSaver: q=2.70857e+06 n=1003226 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11329 XUs and 8377 XVs -> 76 XUVs
C:0 T:1 72775 XUs and 54824 XVs -> 2593 XUVs
C:0 T:2 12929 XUs and 17949 XVs -> 615 XUVs
C:0 T:4 69 XUs and 21 XVs -> 0 XUVs
C:0 T:5 44337 XUs and 10031 XVs -> 453 XUVs
C:0 T:6 12972 XUs and 16365 XVs -> 377 XUVs
4114 XUVs total
1979 collection wire objects
4114 potential space points
Neighbour search...
88938 tests to find 40376 neighbours
Iterating with no regularization...
Begin: 3.58384e+07
0 3.37363e+07
1 3.35751e+07
2 3.35667e+07
Now with regularization...
Begin: 3.2071e+07
0 3.20686e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:40:03 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15482!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 101071 traces, 101071 summary
Retagger: tagging trace set: gauss with 82006 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 82006 traces tagged "gauss"
FrameSaver: q=2.44626e+06 n=1246837 tag=gauss
wclsFrameSaver: saving 101071 traces tagged "wiener"
FrameSaver: q=3.30211e+06 n=1231621 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 17541 XUs and 13226 XVs -> 188 XUVs
C:0 T:1 137196 XUs and 109170 XVs -> 4507 XUVs
C:0 T:2 13918 XUs and 13987 XVs -> 752 XUVs
C:0 T:4 14 XUs and 15 XVs -> 0 XUVs
C:0 T:5 37688 XUs and 31528 XVs -> 517 XUVs
C:0 T:6 10424 XUs and 7692 XVs -> 675 XUVs
C:0 T:7 0 XUs and 0 XVs -> 0 XUVs
6639 XUVs total
2886 collection wire objects
6639 potential space points
Neighbour search...
242403 tests to find 88438 neighbours
Iterating with no regularization...
Begin: 4.39765e+07
0 4.04469e+07
1 4.01341e+07
2 4.01151e+07
Now with regularization...
Begin: 3.83498e+07
0 3.83457e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:41:50 UTC  Closed output file "pdhd_prod_beam__231691_35_1_20251008T183122Z_gen_g4_IonScintPDExt.root_227038_36_1_20251010T161213Z_PDInt.root_234970_15_1_20251029T193736Z_detsim_reco1.root"
29-Oct-2025 20:41:50 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            62.918        89.0917       130.363       88.548        19.5036        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00111689    0.00350084     0.0064807    0.00312802    0.00193626       10     
reco:opdec:Deconvolution                             0.493531      0.588837      0.725163      0.598486      0.0674843       10     
reco:ophitspe:OpHitFinderDeco                        0.0982984     0.104666      0.114421       0.10317     0.00520075       10     
reco:opflash:OpFlashFinder                           0.0440228     0.0681491     0.0911722     0.0681367     0.0120657       10     
reco:opslicer:OpSlicer                               0.123133      0.175144      0.227371       0.18506      0.0324535       10     
reco:wclsdatahd:WireCellToolkit                       31.0775       31.9659       33.0934       32.0326      0.570444        10     
reco:gaushit:GausHitFinder                           0.321315      0.521236      0.685025      0.523354      0.105578        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00017827    0.000287915   0.000527627   0.000250828   0.000105638      10     
reco:reco3d:SpacePointSolver                         0.292926      0.553683      0.986716      0.505487      0.211117        10     
reco:hitpdune:DisambigFromSpacePoints                0.175733      0.456209      0.797705       0.4673       0.178502        10     
reco:pandora:StandardPandora                          6.05327       16.8555       34.6257       16.4597       8.81181        10     
reco:pandoraWriter:StandardPandora                   0.0998148     0.165911      0.253504      0.168355      0.0409429       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.581744       1.13223       2.27878      0.983431      0.473235        10     
reco:pandoraShower:LArPandoraShowerCreation          0.607084       1.07291       2.07165       1.0577       0.434396        10     
reco:pandoracalo:Calorimetry                         0.339844      0.625989       1.2381       0.530506      0.258457        10     
reco:pandoracalonosce:Calorimetry                    0.333899       0.61695       1.21207       0.52221      0.253928        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.629239       1.14903       2.26175       1.04478      0.508483        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.589023       1.09524       2.1364        1.00033      0.478639        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0232232     0.034391      0.0441929     0.0349198    0.00649215       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0230548     0.0330629     0.0424182     0.0331776     0.0059466       10     
reco:emtrkmichelid:EmTrackMichelId                    14.3108       21.3497       28.8408       21.9301       4.31904        10     
reco:crttag:SingleCRTMatchingProducer               0.00262483     0.0040867    0.00757764    0.00361742    0.00149296       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00213602    0.00299753    0.00387206    0.00291718    0.000566127      10     
reco:pandora2Track:LArPandoraTrackCreation            1.52489       2.77204       5.21379       2.77084       1.11537        10     
reco:pandora2calo:ShowerCalorimetry                  0.542476       1.04363       2.03021      0.947455      0.477147        10     
reco:pandora2calonosce:ShowerCalorimetry             0.528037       1.01971       1.99441      0.940857      0.463237        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0238256     0.034381      0.0467965     0.034751     0.00675068       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0247061     0.0343521     0.0453977     0.0349001    0.00684924       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.664349       1.24568       2.33609       1.24129      0.496959        10     
reco:pandora2Showercalo:ShowerCalorimetry             0.74408       1.41304       2.74532        1.368       0.613281        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.720258       1.37731       2.64016       1.34157      0.588309        10     
[art]:TriggerResults:TriggerResultInserter          2.2874e-05    3.32413e-05   9.8037e-05    2.60545e-05   2.16718e-05      10     
end_path:out1:RootOutput                             8.366e-06    1.1608e-05    3.2592e-05    9.2825e-06    7.02954e-06      10     
end_path:out1:RootOutput(write)                       1.12098       1.50945       1.87143       1.52239      0.237109        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6299.86 MB
  Peak resident set size usage (VmHWM): 4097.3 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11788212
-rw-r--r-- 1 dunegli fnalgrid      39091 Oct 29 20:41 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Oct 29 20:41 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:41 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:41 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:41 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:41 time.db
-rw-r--r-- 1 dunegli fnalgrid 6187202799 Oct 29 20:41 pdhd_prod_beam__231691_35_1_20251008T183122Z_gen_g4_IonScintPDExt.root_227038_36_1_20251010T161213Z_PDInt.root_234970_15_1_20251029T193736Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   54056440 Oct 29 20:41 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:27 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:25 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:25 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5829564798 Oct 29 20:25 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-05 04:48:33 UTC       justIN version: 01.05.01