justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.19@dunegpschedd02.fnal.gov

Jobsub ID234970.19@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:51:56
From worker nodeHostnamedunegli-7568266-0-fnpc22003.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:38:18
Input filesusertests:pdhd_prod_beam__226242_49_1_20251008T182836Z_gen_g4_IonScintPDExt.root_227038_84_1_20251010T163610Z_PDInt.root
JobscriptExit code0
Real time1h (4015s)
CPU time1h (3943s = 98%)
Max RSS bytes7573299200 (7222 MiB)
Outputting started2025-10-29 20:45:14
Output files
Finished2025-10-29 20:51:56
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

d. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:41:35 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17213!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 107311 traces, 107311 summary
Retagger: tagging trace set: gauss with 81813 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 81813 traces tagged "gauss"
FrameSaver: q=2.57151e+06 n=1285263 tag=gauss
wclsFrameSaver: saving 107311 traces tagged "wiener"
FrameSaver: q=3.26774e+06 n=1241868 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13799 XUs and 10360 XVs -> 155 XUVs
C:0 T:1 83940 XUs and 64989 XVs -> 1832 XUVs
C:0 T:2 16118 XUs and 13017 XVs -> 227 XUVs
C:0 T:5 38618 XUs and 43196 XVs -> 1120 XUVs
C:0 T:6 40432 XUs and 58955 XVs -> 5002 XUVs
C:0 T:7 33 XUs and 62 XVs -> 20 XUVs
8356 XUVs total
2548 collection wire objects
8356 potential space points
Neighbour search...
361780 tests to find 214498 neighbours
Iterating with no regularization...
Begin: 5.25321e+07
0 4.87599e+07
1 4.83161e+07
2 4.82484e+07
3 4.82295e+07
Now with regularization...
Begin: 4.6446e+07
0 4.64353e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:43:22 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17936!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 109963 traces, 109963 summary
Retagger: tagging trace set: gauss with 88631 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88631 traces tagged "gauss"
FrameSaver: q=2.70466e+06 n=1434522 tag=gauss
wclsFrameSaver: saving 109963 traces tagged "wiener"
FrameSaver: q=3.56879e+06 n=1374757 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14930 XUs and 11881 XVs -> 146 XUVs
C:0 T:1 135912 XUs and 100074 XVs -> 3360 XUVs
C:0 T:2 27230 XUs and 18382 XVs -> 505 XUVs
C:0 T:3 10 XUs and 12 XVs -> 0 XUVs
C:0 T:4 19 XUs and 11 XVs -> 0 XUVs
C:0 T:5 31683 XUs and 28793 XVs -> 451 XUVs
C:0 T:6 70814 XUs and 35807 XVs -> 1706 XUVs
6168 XUVs total
2875 collection wire objects
6168 potential space points
Neighbour search...
215242 tests to find 101866 neighbours
Iterating with no regularization...
Begin: 3.79586e+07
0 3.5131e+07
1 3.4904e+07
2 3.48824e+07
Now with regularization...
Begin: 3.36435e+07
0 3.3636e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:45:13 UTC  Closed output file "pdhd_prod_beam__226242_49_1_20251008T182836Z_gen_g4_IonScintPDExt.root_227038_84_1_20251010T163610Z_PDInt.root_234970_19_1_20251029T193820Z_detsim_reco1.root"
29-Oct-2025 20:45:13 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            55.8614       97.4227       135.111       97.579        20.7458        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000361885   0.00397655    0.00770477    0.00397596    0.00224972       10     
reco:opdec:Deconvolution                              0.48263      0.672731      0.924701      0.682338      0.108998        10     
reco:ophitspe:OpHitFinderDeco                        0.100071      0.108484      0.117024      0.108239      0.0045446       10     
reco:opflash:OpFlashFinder                           0.0390425     0.0828206     0.134416      0.0857856     0.0255284       10     
reco:opslicer:OpSlicer                               0.105659      0.210611      0.324165      0.207006      0.0540504       10     
reco:wclsdatahd:WireCellToolkit                       32.2704       33.0911       33.6265       33.1547      0.351269        10     
reco:gaushit:GausHitFinder                           0.282449      0.580973       0.80499      0.586756      0.140262        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000143633   0.000274187   0.000402513   0.000270057   6.67345e-05      10     
reco:reco3d:SpacePointSolver                         0.186867       0.64447       1.03517      0.679617      0.213841        10     
reco:hitpdune:DisambigFromSpacePoints                0.117891      0.505147      0.983007      0.478374      0.210381        10     
reco:pandora:StandardPandora                          5.66659       16.7867       27.3937       17.8744       6.36128        10     
reco:pandoraWriter:StandardPandora                   0.0927082      0.19289      0.298003      0.184976      0.0553379       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.354279       1.39981       2.60498       1.30079      0.612175        10     
reco:pandoraShower:LArPandoraShowerCreation          0.261034       1.5078        2.63465       1.44239      0.660178        10     
reco:pandoracalo:Calorimetry                         0.194921      0.762419       1.34908      0.691027      0.324538        10     
reco:pandoracalonosce:Calorimetry                     0.19289      0.760741       1.35907      0.680503      0.330335        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.248242       1.57811       2.80108       1.41221      0.783493        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.240752       1.47242       2.55364       1.34997      0.689291        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0173189     0.0369218     0.050553      0.0365326    0.00820239       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0175547     0.0352582     0.0496242     0.0345775    0.00790266       10     
reco:emtrkmichelid:EmTrackMichelId                    12.0163       23.6467       33.2745       24.6364       5.31117        10     
reco:crttag:SingleCRTMatchingProducer               0.00189448    0.00448016    0.00676589    0.00444532    0.00115946       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00168725     0.0035333    0.00463987    0.00358957    0.000813527      10     
reco:pandora2Track:LArPandoraTrackCreation           0.658416       3.60704       6.5867        3.46983       1.5247         10     
reco:pandora2calo:ShowerCalorimetry                  0.209108       1.3573        2.42628       1.23274      0.629736        10     
reco:pandora2calonosce:ShowerCalorimetry             0.198931       1.32993       2.37085       1.21586       0.62372        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0165726     0.0377889     0.0508051     0.0376626    0.00843353       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0160195     0.0363015     0.0496446     0.035977      0.0082058       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.30096       1.65837       2.9049        1.62602      0.698603        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.300221       1.81661       3.34745       1.65113      0.841883        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.289266       1.76541       3.2965        1.60775      0.827225        10     
[art]:TriggerResults:TriggerResultInserter          1.6481e-05    3.24051e-05   0.00012173    2.28425e-05   2.99235e-05      10     
end_path:out1:RootOutput                             3.206e-06    1.00179e-05   3.7721e-05     7.078e-06    9.88275e-06      10     
end_path:out1:RootOutput(write)                      0.933409       1.64586       2.26358       1.66014      0.318801        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6241.24 MB
  Peak resident set size usage (VmHWM): 4045.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12420540
-rw-r--r-- 1 dunegli fnalgrid      39132 Oct 29 20:45 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Oct 29 20:45 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:45 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:45 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:45 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:45 time.db
-rw-r--r-- 1 dunegli fnalgrid 6521532277 Oct 29 20:45 pdhd_prod_beam__226242_49_1_20251008T182836Z_gen_g4_IonScintPDExt.root_227038_84_1_20251010T163610Z_PDInt.root_234970_19_1_20251029T193820Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   58201900 Oct 29 20:44 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:29 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:27 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:27 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6138586455 Oct 29 20:27 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Oct 29 19:38 all-input-dids.txt
justIN time: 2025-11-05 04:48:28 UTC       justIN version: 01.05.01