justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.18@dunegpschedd02.fnal.gov

Jobsub ID234970.18@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:44:37
From worker nodeHostnamedunegli-7568340-0-fnpc23018.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:52
Input filesusertests:pdhd_prod_beam__231691_97_1_20251008T183413Z_gen_g4_IonScintPDExt.root_227038_23_1_20251010T161205Z_PDInt.root
JobscriptExit code0
Real time1h (3602s)
CPU time59m (3540s = 98%)
Max RSS bytes6575915008 (6271 MiB)
Outputting started2025-10-29 20:37:55
Output files
Finished2025-10-29 20:44:37
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e+07
2 2.41914e+07
Now with regularization...
Begin: 2.31183e+07
0 2.3114e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:34:49 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15620!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 112352 traces, 112352 summary
Retagger: tagging trace set: gauss with 88475 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88475 traces tagged "gauss"
FrameSaver: q=2.78752e+06 n=1383201 tag=gauss
wclsFrameSaver: saving 112352 traces tagged "wiener"
FrameSaver: q=3.74098e+06 n=1323809 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 16961 XUs and 12433 XVs -> 146 XUVs
C:0 T:1 109025 XUs and 90673 XVs -> 5072 XUVs
C:0 T:2 77930 XUs and 82979 XVs -> 1397 XUVs
C:0 T:3 27 XUs and 80 XVs -> 0 XUVs
C:0 T:4 7 XUs and 9 XVs -> 0 XUVs
C:0 T:5 36662 XUs and 18596 XVs -> 434 XUVs
C:0 T:6 72611 XUs and 61950 XVs -> 1810 XUVs
8859 XUVs total
2971 collection wire objects
8859 potential space points
Neighbour search...
394313 tests to find 234566 neighbours
Iterating with no regularization...
Begin: 5.3136e+07
0 4.87082e+07
1 4.84276e+07
2 4.83851e+07
Now with regularization...
Begin: 4.65815e+07
0 4.65684e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:36:35 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13106!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 79673 traces, 79673 summary
Retagger: tagging trace set: gauss with 65799 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 65799 traces tagged "gauss"
FrameSaver: q=1.94125e+06 n=1034937 tag=gauss
wclsFrameSaver: saving 79673 traces tagged "wiener"
FrameSaver: q=2.44864e+06 n=1015849 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8692 XUs and 5948 XVs -> 133 XUVs
C:0 T:1 72621 XUs and 52746 XVs -> 3007 XUVs
C:0 T:2 50545 XUs and 108812 XVs -> 949 XUVs
C:0 T:3 9 XUs and 11 XVs -> 3 XUVs
C:0 T:4 9 XUs and 11 XVs -> 0 XUVs
C:0 T:5 25368 XUs and 13226 XVs -> 546 XUVs
C:0 T:6 17523 XUs and 42330 XVs -> 1087 XUVs
C:0 T:7 14 XUs and 12 XVs -> 1 XUVs
5726 XUVs total
2134 collection wire objects
5726 potential space points
Neighbour search...
312688 tests to find 120424 neighbours
Iterating with no regularization...
Begin: 8.2563e+07
0 7.63735e+07
1 7.6115e+07
2 7.60824e+07
Now with regularization...
Begin: 7.48815e+07
0 7.48699e+07
29-Oct-2025 20:37:54 UTC  Closed output file "pdhd_prod_beam__231691_97_1_20251008T183413Z_gen_g4_IonScintPDExt.root_227038_23_1_20251010T161205Z_PDInt.root_234970_18_1_20251029T193754Z_detsim_reco1.root"
29-Oct-2025 20:37:54 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            55.4003        90.39        118.86        91.6644       16.8108        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000387601   0.00411695    0.00755287    0.00453258    0.00221068       10     
reco:opdec:Deconvolution                             0.445544       0.60463      0.781418      0.611336      0.0900235       10     
reco:ophitspe:OpHitFinderDeco                        0.103058      0.115319      0.140905      0.112254      0.0102919       10     
reco:opflash:OpFlashFinder                           0.0382415     0.0642485     0.0828582     0.0717014     0.0154561       10     
reco:opslicer:OpSlicer                               0.110165      0.170219      0.215808      0.172726      0.0289717       10     
reco:wclsdatahd:WireCellToolkit                       32.2323       34.1469       41.1686       33.0471       2.6374         10     
reco:gaushit:GausHitFinder                            0.28874       0.51671      0.676573      0.527236      0.105168        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000121732   0.000278148   0.000508241   0.000241868   0.000115495      10     
reco:reco3d:SpacePointSolver                         0.170178      0.661487       1.01072      0.667219      0.274934        10     
reco:hitpdune:DisambigFromSpacePoints                0.102502      0.405517      0.676453      0.386128      0.188824        10     
reco:pandora:StandardPandora                          3.0504        15.1054       27.2709       14.5143       6.90929        10     
reco:pandoraWriter:StandardPandora                   0.0803402     0.164567      0.240517      0.168888      0.0427502       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.380655      0.990033       1.65559      0.925599      0.366279        10     
reco:pandoraShower:LArPandoraShowerCreation          0.391605       1.09976       1.8246        1.04511      0.397618        10     
reco:pandoracalo:Calorimetry                         0.210466      0.559815      0.923568      0.513408      0.207157        10     
reco:pandoracalonosce:Calorimetry                    0.212293      0.552104      0.921016      0.501394      0.206522        10     
reco:pandoraShowercalo:ShowerCalorimetry              0.38714       1.17038       1.98256       1.14761      0.452541        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.369509       1.11063       1.89629       1.08651      0.434276        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0196047     0.0337809     0.0430916     0.0336926    0.00713216       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0195348     0.0317567     0.0407706     0.032675     0.00627456       10     
reco:emtrkmichelid:EmTrackMichelId                    12.6963       22.471        28.1276       23.2206       4.30819        10     
reco:crttag:SingleCRTMatchingProducer               0.00234435    0.00400019    0.00829085    0.00368972    0.00154922       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00196931    0.00305569    0.00397659    0.00309433    0.000544434      10     
reco:pandora2Track:LArPandoraTrackCreation           0.842164       2.68283       4.79884       2.51047       1.03752        10     
reco:pandora2calo:ShowerCalorimetry                  0.324298       1.03867       1.86088      0.995201       0.41866        10     
reco:pandora2calonosce:ShowerCalorimetry             0.302596       1.01659       1.8457        0.9691       0.419375        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0196153     0.0337651     0.0428737     0.0333111    0.00687057       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0187358     0.031545      0.0412899     0.0321641    0.00634308       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.376182        1.216        2.20633       1.10321      0.481827        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.397208       1.36803       2.52381       1.25759      0.580627        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.383746       1.33576       2.47937       1.22664      0.569638        10     
[art]:TriggerResults:TriggerResultInserter          2.1041e-05    3.1867e-05    8.4722e-05    2.6055e-05    1.78922e-05      10     
end_path:out1:RootOutput                             7.956e-06    1.08798e-05   2.8354e-05     9.288e-06    5.85179e-06      10     
end_path:out1:RootOutput(write)                       1.29326       1.6244        1.93448       1.6378       0.216181        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6207.93 MB
  Peak resident set size usage (VmHWM): 3994.64 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10970176
-rw-r--r-- 1 dunegli fnalgrid      38762 Oct 29 20:37 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Oct 29 20:37 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:37 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:37 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:37 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:37 time.db
-rw-r--r-- 1 dunegli fnalgrid 5761109168 Oct 29 20:37 pdhd_prod_beam__231691_97_1_20251008T183413Z_gen_g4_IonScintPDExt.root_227038_23_1_20251010T161205Z_PDInt.root_234970_18_1_20251029T193754Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51477050 Oct 29 20:37 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:21 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:21 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5420569564 Oct 29 20:21 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-05 04:48:30 UTC       justIN version: 01.05.01