justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 248452.1@dunegpschedd02.fnal.gov

Jobsub ID248452.1@dunegpschedd02.fnal.gov
Workflow ID10372
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-19 09:33:44
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-11-19 13:04:12
From worker nodeHostnametds504.pic.es
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-19 09:35:01
Input filesmonte-carlo-010372-000001
JobscriptExit code0
Real time3h (12504s)
CPU time3h (11819s = 94%)
Max RSS bytes4044865536 (3857 MiB)
Outputting started2025-11-19 13:03:27
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10372/1/001/reco_stage1.root
Finished2025-11-19 13:04:12
Saved logsjustin-logs:248452.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e:prepare:   Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 118169 traces, 0 summary
Retagger: tagging trace set: gauss with 91695 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 91695 traces tagged "gauss"
FrameSaver: q=3.30911e+06 n=1644246 tag=gauss
wclsFrameSaver: saving 118169 traces tagged "wiener"
FrameSaver: q=3.48046e+06 n=1459052 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 49727 XUs and 36930 XVs -> 2529 XUVs
C:0 T:1 109163 XUs and 75091 XVs -> 6623 XUVs
C:0 T:2 27415 XUs and 46683 XVs -> 1914 XUVs
C:0 T:3 14202 XUs and 25514 XVs -> 84 XUVs
C:0 T:4 15527 XUs and 13870 XVs -> 137 XUVs
C:0 T:5 81557 XUs and 47795 XVs -> 10174 XUVs
C:0 T:6 55598 XUs and 71965 XVs -> 2271 XUVs
C:0 T:7 24942 XUs and 33414 XVs -> 239 XUVs
C:0 T:8 9193 XUs and 6756 XVs -> 50 XUVs
C:0 T:9 68895 XUs and 60862 XVs -> 5152 XUVs
C:0 T:10 20356 XUs and 13652 XVs -> 2167 XUVs
C:0 T:11 89 XUs and 54 XVs -> 7 XUVs
31347 XUVs total
10333 collection wire objects
27503 potential space points
Neighbour search...
1968577 tests to find 1048756 neighbours
Iterating with no regularization...
Begin: 7.4953e+07
0 6.45481e+07
1 6.37374e+07
2 6.36216e+07
3 6.3592e+07
Now with regularization...
Begin: 5.33986e+07
0 5.33613e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410607224
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410607224
fLifetime: 30000 [us]
19-Nov-2025 14:03:24 CET  Closed output file "reco_stage1.root"
19-Nov-2025 14:03:24 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          274.586       404.862       527.325       399.131       87.3657        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00131873    0.00396832    0.00666372    0.00444442    0.00162059       10     
reco:ophit:OpHitFinder                             0.310809      0.356697      0.428016      0.343581      0.0387389       10     
reco:opflash:OpFlashFinder                         0.117111      0.166812      0.215633      0.159707      0.0350991       10     
reco:opslicer:OpSlicer                             0.665977      0.837204       1.05861      0.836457      0.112325        10     
reco:caldata:DataPrepModule                         5.40856       7.44422       7.81953       7.66907      0.683738        10     
reco:wclsdatasp:WireCellToolkit                     39.3518       43.7091       48.4678       43.5141       2.67572        10     
reco:gaushit:GausHitFinder                         0.905739       1.48244       2.20692       1.46028       0.3661         10     
reco:nhitsfilter:NumberOfHitsFilter               0.000439198   0.000826662   0.00146744    0.00075838    0.000328852      10     
reco:reco3d:SpacePointSolver                        6.35859       17.4746       27.6712       14.2173       7.84019        10     
reco:hitpdune:DisambigFromSpacePoints              0.927243       1.78491       2.81256       1.77576      0.591105        10     
reco:pandora:StandardPandora                        16.4192       30.7277       44.4895       30.0807       8.69048        10     
reco:pandoraWriter:StandardPandora                 0.310097      0.587928      0.758704      0.571111      0.140758        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.93053       5.44114       8.21361       5.58822       1.84919        10     
reco:pandoraShower:LArPandoraShowerCreation         2.27794       6.20408       10.889        6.39467       2.50193        10     
reco:pandoracalo:Calorimetry                       0.994185       3.28765       7.35338       2.95661       1.70884        10     
reco:pandoracalonosce:Calorimetry                  0.993269       3.08027       4.93101       2.98875       1.29719        10     
reco:pandorapid:Chi2ParticleID                    0.00509782     0.0071648     0.0086261    0.00719875    0.00113607       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0320218     0.0721373     0.0888422     0.0781478     0.0173669       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00361804     0.0069494     0.0092097    0.00709308    0.00152502       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.25666       7.04548       13.1305       6.97992       3.35501        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.51672       6.93249        13.7         6.79034       3.39402        10     
reco:emtrkmichelid:EmTrackMichelId                  154.403       208.989       264.074       210.122       36.2272        10     
reco:crttag:SingleCRTMatchingProducer              0.0113692     0.0200459     0.0294311     0.0203349    0.00584253       10     
reco:crtreco:TwoCRTMatchingProducer               0.00707435     0.0119752     0.0188342     0.010695     0.00335001       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.710508      0.934013       1.31504      0.925203       0.19384        10     
reco:pandora2Track:LArPandoraTrackCreation          7.47879       17.0141       26.8343       17.9229       6.86772        10     
reco:pandora2calo:Calorimetry                       3.01115       8.8215        13.4121       9.83352       3.57802        10     
reco:pandora2calonosce:Calorimetry                  3.35126       8.61329       14.214        8.81302       3.67595        10     
reco:pandora2pid:Chi2ParticleID                    0.0103177     0.0165695     0.0273574     0.0160126    0.00481718       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.053094      0.0860019     0.130649      0.0810544     0.0221177       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0103704     0.016262      0.0255282     0.0156217    0.00425068       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.11977       6.12081       10.3336       5.52559       2.72309        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.01001       7.44194       11.5064       6.73936       3.01302        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.24937       6.62761       10.4138       5.52307       2.9205         10     
[art]:TriggerResults:TriggerResultInserter         5.819e-05    9.22722e-05   0.00018568    8.86275e-05   3.40082e-05      10     
end_path:out1:RootOutput                          1.0429e-05    1.7534e-05    2.4316e-05    1.6216e-05    4.21375e-06      10     
end_path:out1:RootOutput(write)                     2.25564       3.29034       4.37665       3.42437      0.708965        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4879.56 MB
  Peak resident set size usage (VmHWM): 4044.87 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.65 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 8998640
-rw-r--r-- 1 dune001 dune 2150800063 Nov 19 12:53 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2042828860 Nov 19 12:51 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 1808529155 Nov 19 14:03 reco_stage1.root
-rw-r--r-- 1 dune001 dune 1595678886 Nov 19 10:54 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1548522822 Nov 19 10:43 g4_stage1.root
-rw-r--r-- 1 dune001 dune   67615590 Nov 19 13:58 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     242615 Nov 19 14:03 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Nov 19 14:03 mem.db
-rw-r--r-- 1 dune001 dune      62376 Nov 19 14:03 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Nov 19 14:03 time.db
-rw-r--r-- 1 dune001 dune        519 Nov 19 12:53 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Nov 19 10:54 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Nov 19 14:03 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Nov 19 10:35 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Nov 19 14:03 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Nov 19 12:58 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Nov 19 12:53 debugprod.log
justIN time: 2026-02-09 04:59:46 UTC       justIN version: 01.06.00