justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 270887.0@dunegpschedd01.fnal.gov

Jobsub ID270887.0@dunegpschedd01.fnal.gov
Workflow ID11411
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-30 06:20:17
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce04
Last heartbeat2025-12-30 09:43:41
From worker nodeHostnameccwcondor0375
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-30 06:20:55
Input filesmonte-carlo-011411-000001
JobscriptExit code0
Real time3h (12080s)
CPU time3h (11704s = 96%)
Max RSS bytes4036567040 (3849 MiB)
Outputting started2025-12-30 09:42:16
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11411/1/001/reco_stage1.root
Finished2025-12-30 09:43:41
Saved logsjustin-logs:270887.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 125717 traces, 0 summary
Retagger: tagging trace set: gauss with 93235 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 93235 traces tagged "gauss"
FrameSaver: q=3.22327e+06 n=1623646 tag=gauss
wclsFrameSaver: saving 125717 traces tagged "wiener"
FrameSaver: q=3.43068e+06 n=1420481 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 29532 XUs and 37743 XVs -> 375 XUVs
C:0 T:1 71161 XUs and 140254 XVs -> 11716 XUVs
C:0 T:2 89806 XUs and 97910 XVs -> 3984 XUVs
C:0 T:3 44808 XUs and 42236 XVs -> 285 XUVs
C:0 T:4 7029 XUs and 11686 XVs -> 77 XUVs
C:0 T:5 39755 XUs and 86463 XVs -> 8871 XUVs
C:0 T:6 133156 XUs and 109213 XVs -> 6658 XUVs
C:0 T:7 47707 XUs and 38407 XVs -> 386 XUVs
C:0 T:8 4569 XUs and 5828 XVs -> 50 XUVs
C:0 T:9 22808 XUs and 41346 XVs -> 2412 XUVs
C:0 T:10 100330 XUs and 56466 XVs -> 15976 XUVs
C:0 T:11 144 XUs and 134 XVs -> 11 XUVs
50801 XUVs total
11797 collection wire objects
47604 potential space points
Neighbour search...
7569124 tests to find 3419176 neighbours
Iterating with no regularization...
Begin: 8.01223e+07
0 5.9123e+07
1 5.797e+07
2 5.77936e+07
3 5.77354e+07
4 5.771e+07
Now with regularization...
Begin: 4.79463e+07
0 4.78902e+07
1 4.78516e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411429323
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411429323
fLifetime: 30000 [us]
30-Dec-2025 10:42:14 CET  Closed output file "reco_stage1.root"
30-Dec-2025 10:42:14 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          268.265       370.589       545.426       364.203       77.7574        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000997617   0.00455718    0.00615089    0.00530635    0.00171116       10     
reco:ophit:OpHitFinder                             0.219647      0.303215      0.351676      0.304292      0.0378213       10     
reco:opflash:OpFlashFinder                         0.126073      0.170784      0.218497      0.170565      0.0300314       10     
reco:opslicer:OpSlicer                             0.386799      0.684442      0.908779      0.688802      0.148647        10     
reco:caldata:DataPrepModule                         4.84863       5.89163       6.1855        5.95258      0.361258        10     
reco:wclsdatasp:WireCellToolkit                     31.2497       34.8319       38.6582       34.7419       2.04039        10     
reco:gaushit:GausHitFinder                          1.05872       1.31998       1.64326       1.2738       0.209548        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000256876   0.000514981   0.000782434   0.000502053   0.000150941      10     
reco:reco3d:SpacePointSolver                        6.31531       16.5884       29.1656       16.8001       6.21892        10     
reco:hitpdune:DisambigFromSpacePoints              0.968276       1.61663       2.12128       1.71967      0.414865        10     
reco:pandora:StandardPandora                        15.6267       31.8663       58.454        31.7219       13.149         10     
reco:pandoraWriter:StandardPandora                 0.396567      0.530418      0.698331      0.539959      0.0872097       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.6914        5.33449       9.84502       4.8133        2.03742        10     
reco:pandoraShower:LArPandoraShowerCreation         2.59788       5.07945       10.0182       4.78905       2.03727        10     
reco:pandoracalo:Calorimetry                        1.4645        2.97977       5.17218       2.77699       1.04218        10     
reco:pandoracalonosce:Calorimetry                   1.32602       2.73402       4.48372       2.8018        1.03271        10     
reco:pandorapid:Chi2ParticleID                    0.00419683     0.0064557    0.00865912     0.0066893    0.00124312       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0531523     0.0666378     0.0820948     0.0671994    0.00951359       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00419247    0.00602281    0.00734281    0.00582183    0.000978877      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.65268       5.55427       12.0912       5.28989       2.51983        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.33686       5.03845       10.8134       4.62066       2.28671        10     
reco:emtrkmichelid:EmTrackMichelId                  156.37        198.837       263.537       195.789       30.9241        10     
reco:crttag:SingleCRTMatchingProducer              0.0134794     0.0217076     0.0526405     0.0173725     0.0114409       10     
reco:crtreco:TwoCRTMatchingProducer               0.00674537     0.0136504     0.0321583     0.0117176     0.0067798       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.613751      0.879447       1.09465      0.920617      0.146981        10     
reco:pandora2Track:LArPandoraTrackCreation          6.60947       15.5353       27.9838       15.6353       5.51954        10     
reco:pandora2calo:Calorimetry                       3.27607       7.92245       16.1706       7.61461       3.39907        10     
reco:pandora2calonosce:Calorimetry                  2.84488       7.1438        14.4042       6.9285        3.05367        10     
reco:pandora2pid:Chi2ParticleID                   0.00824261     0.0141421     0.0216163     0.0144609    0.00358057       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.045727      0.0802262     0.105262      0.0831798     0.0150228       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0101148     0.013233      0.0165244     0.0130988    0.00206968       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.72427       5.13086       10.0099       4.89252       1.89491        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.7869        6.23968       11.7296       5.97106       2.29426        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.68756       5.16952       8.10155       5.29654       1.5475         10     
[art]:TriggerResults:TriggerResultInserter        5.4342e-05    9.8489e-05    0.000267612   7.5844e-05    6.04869e-05      10     
end_path:out1:RootOutput                          1.0827e-05    1.56895e-05   3.3631e-05    1.37555e-05   6.24921e-06      10     
end_path:out1:RootOutput(write)                     2.28687       2.79539       3.38371       2.71033      0.393608        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4879.54 MB
  Peak resident set size usage (VmHWM): 4036.57 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.61 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9844344
-rw-r--r-- 1 dune001 lbno 2316933880 Dec 30 09:38 detsim_stage2.root
-rw-r--r-- 1 dune001 lbno 2197180311 Dec 30 09:36 detsim_stage1.root
-rw-r--r-- 1 dune001 lbno 2001713023 Dec 30 10:42 reco_stage1.root
-rw-r--r-- 1 dune001 lbno 1785968732 Dec 30 07:41 g4_stage2.root
-rw-r--r-- 1 dune001 lbno 1703975349 Dec 30 07:28 g4_stage1.root
-rw-r--r-- 1 dune001 lbno   74220980 Dec 30 10:38 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno     242600 Dec 30 10:42 jobscript.log
-rw-r--r-- 1 dune001 lbno     217088 Dec 30 10:42 mem.db
-rw-r--r-- 1 dune001 lbno      62376 Dec 30 10:42 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno      36864 Dec 30 10:42 time.db
-rw-r--r-- 1 dune001 lbno        519 Dec 30 09:38 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 lbno        519 Dec 30 07:41 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 lbno        519 Dec 30 10:42 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 lbno         52 Dec 30 07:20 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno          7 Dec 30 10:42 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Dec 30 09:44 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Dec 30 09:38 debugprod.log
justIN time: 2026-02-07 01:00:44 UTC       justIN version: 01.06.00