justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265520.1@dunegpschedd02.fnal.gov

Jobsub ID265520.1@dunegpschedd02.fnal.gov
Workflow ID11470
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-02 09:24:17
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2026-01-02 15:18:36
From worker nodeHostnameminis10.farm.particle.cz
cpuinfoAMD EPYC 7351P 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-02 09:51:51
Input filesmonte-carlo-011470-000001
JobscriptExit code0
Real time5h (19554s)
CPU time5h (19000s = 97%)
Max RSS bytes4476456960 (4269 MiB)
Outputting started2026-01-02 15:17:46
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11470/1/001/reco_stage1.root
Finished2026-01-02 15:18:36
Saved logsjustin-logs:265520.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

epService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 133572 traces, 0 summary
Retagger: tagging trace set: gauss with 98687 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 98687 traces tagged "gauss"
FrameSaver: q=3.48986e+06 n=1686544 tag=gauss
wclsFrameSaver: saving 133572 traces tagged "wiener"
FrameSaver: q=3.721e+06 n=1475078 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 29290 XUs and 27734 XVs -> 625 XUVs
C:0 T:1 97737 XUs and 64613 XVs -> 15285 XUVs
C:0 T:2 124317 XUs and 116036 XVs -> 12218 XUVs
C:0 T:3 45349 XUs and 41273 XVs -> 610 XUVs
C:0 T:4 18427 XUs and 16603 XVs -> 260 XUVs
C:0 T:5 75581 XUs and 60897 XVs -> 6597 XUVs
C:0 T:6 52095 XUs and 55012 XVs -> 3137 XUVs
C:0 T:7 21638 XUs and 23287 XVs -> 196 XUVs
C:0 T:8 9573 XUs and 7863 XVs -> 111 XUVs
C:0 T:9 63210 XUs and 64809 XVs -> 5460 XUVs
C:0 T:10 38096 XUs and 18248 XVs -> 4508 XUVs
C:0 T:11 104 XUs and 63 XVs -> 8 XUVs
49015 XUVs total
11827 collection wire objects
44864 potential space points
Neighbour search...
6769378 tests to find 2864306 neighbours
Iterating with no regularization...
Begin: 5.56685e+07
0 4.58799e+07
1 4.5098e+07
2 4.49758e+07
3 4.49415e+07
Now with regularization...
Begin: 3.4588e+07
0 3.45518e+07
1 3.45283e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411491988
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411491988
fLifetime: 30000 [us]
02-Jan-2026 16:17:43 CET  Closed output file "reco_stage1.root"
02-Jan-2026 16:17:43 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          521.027       652.761       865.774       645.203       92.5779        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00114834     0.0053743    0.00732855    0.00645605     0.0021516       10     
reco:ophit:OpHitFinder                             0.321416      0.443761       0.49101      0.467669      0.0518303       10     
reco:opflash:OpFlashFinder                         0.212746      0.255767      0.329024      0.242304      0.0376784       10     
reco:opslicer:OpSlicer                             0.808771       1.19718       1.63295       1.14074      0.244636        10     
reco:caldata:DataPrepModule                         6.08566       7.70179       8.74999       8.04971      0.805425        10     
reco:wclsdatasp:WireCellToolkit                     44.3559       54.5787       62.6413       55.2148       5.1693         10     
reco:gaushit:GausHitFinder                          1.28762       1.99846       2.42552       2.00507      0.323587        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000588953   0.000779657   0.00139231    0.000724061   0.000217452      10     
reco:reco3d:SpacePointSolver                        20.058        24.5897       31.476        23.1393       3.75138        10     
reco:hitpdune:DisambigFromSpacePoints               2.33691       3.08753       4.59137       2.88582      0.728827        10     
reco:pandora:StandardPandora                        43.8566       74.668        142.685       65.537        27.0766        10     
reco:pandoraWriter:StandardPandora                 0.576927      0.745237       1.01675      0.701325      0.150349        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.07503       8.2163        11.3303       7.95275       1.51111        10     
reco:pandoraShower:LArPandoraShowerCreation         7.30318       11.1195       17.221        9.87624       3.38146        10     
reco:pandoracalo:Calorimetry                        3.29708       4.38935       5.88729       4.21173      0.867721        10     
reco:pandoracalonosce:Calorimetry                   2.91515       4.33787       5.9733        4.26429      0.869985        10     
reco:pandorapid:Chi2ParticleID                    0.00631564     0.010065       0.01299      0.0108916     0.0024753       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0611509     0.0930279     0.121315      0.0944551     0.0192124       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00601841    0.00932584     0.0119167    0.00924654    0.00224729       10     
reco:pandoraShowercalo:ShowerCalorimetry            7.09584       10.9821       17.2115       10.407        3.18434        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.08677       10.6829       17.3029       10.4583       2.87043        10     
reco:emtrkmichelid:EmTrackMichelId                  253.455       333.848       406.312       339.084       40.826         10     
reco:crttag:SingleCRTMatchingProducer              0.0172784     0.0261082     0.032525      0.0273065    0.00482153       10     
reco:crtreco:TwoCRTMatchingProducer                0.0127855     0.0182495     0.0215757     0.0180458     0.0025468       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.24978       1.45683       1.68586       1.45925      0.146715        10     
reco:pandora2Track:LArPandoraTrackCreation          21.3957       30.1064       43.5971       28.2551       7.63501        10     
reco:pandora2calo:Calorimetry                       10.4839       14.2344       21.1183       13.1184       3.41408        10     
reco:pandora2calonosce:Calorimetry                  8.89369       13.7937       20.8273       13.5374       3.28355        10     
reco:pandora2pid:Chi2ParticleID                    0.0178019     0.0274433     0.0374581     0.0265967    0.00632005       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0777538     0.130565      0.174107      0.126379      0.0310892       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0176012     0.0269097     0.0338582     0.0281068     0.0055295       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.44678       11.2378       15.4935       10.6196       2.64636        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.8167        12.5781       16.6214       11.6906       2.88139        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.66294       11.0571       13.627        11.3626       2.32615        10     
[art]:TriggerResults:TriggerResultInserter        6.1249e-05    0.000103769   0.000175541   8.72595e-05   3.98553e-05      10     
end_path:out1:RootOutput                           9.187e-06    1.55942e-05   3.6882e-05    1.26795e-05   7.82048e-06      10     
end_path:out1:RootOutput(write)                     3.56388       4.95545       6.86597       4.96323      0.819058        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5282.98 MB
  Peak resident set size usage (VmHWM): 4476.46 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.70 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11553924
-rw-r--r--. 1 fermilab users 2661605066 Jan  2 14:25 detsim_stage2.root
-rw-r--r--. 1 fermilab users 2525282395 Jan  2 14:21 detsim_stage1.root
-rw-r--r--. 1 fermilab users 2410502026 Jan  2 16:17 reco_stage1.root
-rw-r--r--. 1 fermilab users 2116498263 Jan  2 11:24 g4_stage2.root
-rw-r--r--. 1 fermilab users 2026705526 Jan  2 11:01 g4_stage1.root
-rw-r--r--. 1 fermilab users   89992340 Jan  2 16:10 Pandora_Events.pndr
-rw-r--r--. 1 fermilab users     243047 Jan  2 16:17 jobscript.log
-rw-r--r--. 1 fermilab users     217088 Jan  2 16:17 mem.db
-rw-r--r--. 1 fermilab users      62376 Jan  2 16:17 Pandora_Geometry.xml
-rw-r--r--. 1 fermilab users      36864 Jan  2 16:17 time.db
-rw-r--r--. 1 fermilab users        519 Jan  2 14:25 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 fermilab users        519 Jan  2 11:24 g4_protoDUNE_hist.root
-rw-r--r--. 1 fermilab users        519 Jan  2 16:17 protoDUNE_reco1_hist.root
-rw-r--r--. 1 fermilab users         52 Jan  2 10:51 all-input-dids.txt
-rw-r--r--. 1 fermilab users          7 Jan  2 16:17 justin-processed-pfns.txt
-rw-r--r--. 1 fermilab users          0 Jan  2 14:35 ProtoDUNE.log
-rw-r--r--. 1 fermilab users          0 Jan  2 14:25 debugprod.log
justIN time: 2026-02-06 22:22:19 UTC       justIN version: 01.06.00