justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.23@dunegpschedd02.fnal.gov

Jobsub ID234970.23@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:50:51
From worker nodeHostnamedunegli-7568267-0-fnpc23023.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:57
Input filesusertests:pdhd_prod_beam__231691_105_1_20251008T183505Z_gen_g4_IonScintPDExt.root_227038_33_1_20251010T161131Z_PDInt.root
JobscriptExit code0
Real time1h (3968s)
CPU time1h (3882s = 97%)
Max RSS bytes7148548096 (6817 MiB)
Outputting started2025-10-29 20:44:06
Output files
Finished2025-10-29 20:50:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

neighbours
Iterating with no regularization...
Begin: 3.80688e+07
0 3.43617e+07
1 3.42047e+07
2 3.4192e+07
Now with regularization...
Begin: 3.29503e+07
0 3.29466e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:40:20 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16132!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 92462 traces, 92462 summary
Retagger: tagging trace set: gauss with 75395 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75395 traces tagged "gauss"
FrameSaver: q=2.15585e+06 n=1179362 tag=gauss
wclsFrameSaver: saving 92462 traces tagged "wiener"
FrameSaver: q=2.81897e+06 n=1144632 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 16368 XUs and 12800 XVs -> 228 XUVs
C:0 T:1 92834 XUs and 65404 XVs -> 3113 XUVs
C:0 T:2 25183 XUs and 23926 XVs -> 481 XUVs
C:0 T:3 0 XUs and 1 XVs -> 0 XUVs
C:0 T:4 1 XUs and 6 XVs -> 0 XUVs
C:0 T:5 17570 XUs and 23081 XVs -> 2771 XUVs
C:0 T:6 35499 XUs and 37351 XVs -> 1348 XUVs
7941 XUVs total
2397 collection wire objects
7941 potential space points
Neighbour search...
465771 tests to find 275960 neighbours
Iterating with no regularization...
Begin: 3.50925e+07
0 3.2106e+07
1 3.1905e+07
2 3.1871e+07
3 3.18602e+07
Now with regularization...
Begin: 3.07508e+07
0 3.07436e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:41:47 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13629!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 113907 traces, 113907 summary
Retagger: tagging trace set: gauss with 91104 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 91104 traces tagged "gauss"
FrameSaver: q=3.09197e+06 n=1423160 tag=gauss
wclsFrameSaver: saving 113907 traces tagged "wiener"
FrameSaver: q=4.30903e+06 n=1417210 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 20769 XUs and 16398 XVs -> 313 XUVs
C:0 T:1 267098 XUs and 227056 XVs -> 11200 XUVs
C:0 T:2 29549 XUs and 35415 XVs -> 970 XUVs
C:0 T:3 0 XUs and 3 XVs -> 0 XUVs
C:0 T:4 110 XUs and 208 XVs -> 81 XUVs
C:0 T:5 89132 XUs and 70872 XVs -> 8187 XUVs
C:0 T:6 40442 XUs and 40510 XVs -> 1103 XUVs
21854 XUVs total
4925 collection wire objects
21854 potential space points
Neighbour search...
3231394 tests to find 1289034 neighbours
Iterating with no regularization...
Begin: 8.68202e+07
0 7.88023e+07
1 7.81368e+07
2 7.80601e+07
Now with regularization...
Begin: 7.49784e+07
0 7.49535e+07
29-Oct-2025 20:44:05 UTC  Closed output file "pdhd_prod_beam__231691_105_1_20251008T183505Z_gen_g4_IonScintPDExt.root_227038_33_1_20251010T161131Z_PDInt.root_234970_23_1_20251029T193759Z_detsim_reco1.root"
29-Oct-2025 20:44:05 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            69.8308       97.301        134.515       98.1177       19.2676        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000583567   0.00410475    0.00663967     0.0046213    0.00196616       10     
reco:opdec:Deconvolution                             0.476772      0.613925      0.837872      0.588122      0.0977745       10     
reco:ophitspe:OpHitFinderDeco                        0.0978097     0.113361      0.190868      0.103125      0.0264704       10     
reco:opflash:OpFlashFinder                           0.0375286     0.0736854     0.138331       0.06956      0.0262002       10     
reco:opslicer:OpSlicer                               0.136019      0.176974      0.264734      0.162305      0.0389322       10     
reco:wclsdatahd:WireCellToolkit                       31.3443       32.2095       33.0971       32.1882      0.482071        10     
reco:gaushit:GausHitFinder                           0.357465      0.540399      0.685989      0.533223      0.100014        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000183641   0.000302492   0.000591502   0.000245415   0.000129059      10     
reco:reco3d:SpacePointSolver                         0.351909       0.81399       2.19205      0.561072       0.53686        10     
reco:hitpdune:DisambigFromSpacePoints                0.227464      0.527626       1.03112      0.504722      0.206964        10     
reco:pandora:StandardPandora                          9.30021       20.6404       42.5335       18.886        9.84424        10     
reco:pandoraWriter:StandardPandora                   0.115695      0.180974      0.254107      0.178452      0.0423554       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.636085       1.35769       2.04005       1.34464      0.414876        10     
reco:pandoraShower:LArPandoraShowerCreation          0.599054       1.28636       1.90474       1.35969      0.427777        10     
reco:pandoracalo:Calorimetry                         0.348719      0.716033       1.06165      0.673944      0.214293        10     
reco:pandoracalonosce:Calorimetry                    0.342617      0.703258       1.05991      0.661603      0.210945        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.606376       1.3345        2.04816       1.32355      0.485536        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.589172       1.2789        1.93645       1.27292      0.461994        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0237456     0.0341165     0.0468303     0.0328015    0.00694917       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0230054     0.0326619     0.0452119     0.0319557    0.00618812       10     
reco:emtrkmichelid:EmTrackMichelId                    17.0972       22.7472       28.9782       22.465        3.73825        10     
reco:crttag:SingleCRTMatchingProducer               0.00339075    0.00433978    0.00742436    0.00394892    0.00117522       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00267749    0.00317686     0.0040779    0.00303613    0.000459946      10     
reco:pandora2Track:LArPandoraTrackCreation            1.42391       3.24956       5.01492       3.1694        1.10611        10     
reco:pandora2calo:ShowerCalorimetry                   0.52446       1.17101       1.78644       1.16591      0.443384        10     
reco:pandora2calonosce:ShowerCalorimetry             0.502683        1.147        1.78158       1.13757      0.445603        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.023739      0.034256      0.0451813     0.033897     0.00655346       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0233231     0.0326714     0.0440752     0.0320489    0.00630544       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.69094       1.43735       2.21713       1.48149      0.486731        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.706949       1.5929        2.51257       1.52666      0.585365        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.688424       1.55454       2.48174       1.48919      0.577097        10     
[art]:TriggerResults:TriggerResultInserter          2.2303e-05    3.68757e-05   0.000127033   2.67615e-05   3.02326e-05      10     
end_path:out1:RootOutput                             7.264e-06    1.22886e-05   3.8694e-05     9.263e-06    8.86597e-06      10     
end_path:out1:RootOutput(write)                       1.18898       1.61071       2.46162       1.51714      0.328868        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6162.01 MB
  Peak resident set size usage (VmHWM): 3969.17 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12351624
-rw-r--r-- 1 dunegli fnalgrid      38994 Oct 29 20:44 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:44 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:44 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:44 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:44 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:44 time.db
-rw-r--r-- 1 dunegli fnalgrid 6478188233 Oct 29 20:44 pdhd_prod_beam__231691_105_1_20251008T183505Z_gen_g4_IonScintPDExt.root_227038_33_1_20251010T161131Z_PDInt.root_234970_23_1_20251029T193759Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   56639900 Oct 29 20:43 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:28 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:26 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:26 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6112920117 Oct 29 20:26 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-05 04:48:27 UTC       justIN version: 01.05.01