justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 277472.0@dunegpschedd01.fnal.gov

Jobsub ID277472.0@dunegpschedd01.fnal.gov
Workflow ID11862
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 21:59:53
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2026-01-14 03:14:36
From worker nodeHostnamewn-la-23.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 22:01:44
Input filesmonte-carlo-011862-000001
JobscriptExit code0
Real time5h (18673s)
CPU time5h (18550s = 99%)
Max RSS bytes4550193152 (4339 MiB)
Outputting started2026-01-14 03:12:58
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11862/1/001/reco_stage1.root
Finished2026-01-14 03:14:36
Saved logsjustin-logs:277472.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 140180 traces, 0 summary
Retagger: tagging trace set: gauss with 109305 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 109305 traces tagged "gauss"
FrameSaver: q=3.86857e+06 n=1992005 tag=gauss
wclsFrameSaver: saving 140180 traces tagged "wiener"
FrameSaver: q=4.07089e+06 n=1771563 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 40111 XUs and 33604 XVs -> 428 XUVs
C:0 T:1 80060 XUs and 70060 XVs -> 4338 XUVs
C:0 T:2 167202 XUs and 120700 XVs -> 10205 XUVs
C:0 T:3 57036 XUs and 46525 XVs -> 428 XUVs
C:0 T:4 12238 XUs and 9817 XVs -> 253 XUVs
C:0 T:5 56893 XUs and 31053 XVs -> 3659 XUVs
C:0 T:6 129496 XUs and 114498 XVs -> 5550 XUVs
C:0 T:7 58598 XUs and 49469 XVs -> 539 XUVs
C:0 T:8 8427 XUs and 6424 XVs -> 59 XUVs
C:0 T:9 55582 XUs and 59441 XVs -> 4485 XUVs
C:0 T:10 39292 XUs and 29433 XVs -> 3460 XUVs
C:0 T:11 73 XUs and 37 XVs -> 2 XUVs
33406 XUVs total
13176 collection wire objects
30044 potential space points
Neighbour search...
1778350 tests to find 902664 neighbours
Iterating with no regularization...
Begin: 8.82631e+07
0 7.1807e+07
1 7.08738e+07
2 7.07176e+07
3 7.06675e+07
Now with regularization...
Begin: 5.95532e+07
0 5.95069e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411723954
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411723954
fLifetime: 30000 [us]
14-Jan-2026 04:12:56 CET  Closed output file "reco_stage1.root"
14-Jan-2026 04:12:56 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          394.386        552.8        762.626       535.011       110.429        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00117049    0.00602845    0.00977603    0.00632441    0.00292757       10     
reco:ophit:OpHitFinder                             0.412438      0.463411       0.51091      0.470275      0.0319347       10     
reco:opflash:OpFlashFinder                         0.290115       0.35963      0.398982      0.360242      0.0304387       10     
reco:opslicer:OpSlicer                             0.767841      0.920183       1.08971      0.906365      0.0984227       10     
reco:caldata:DataPrepModule                         6.32265       6.59136       7.12785       6.54796      0.204961        10     
reco:wclsdatasp:WireCellToolkit                     44.7971       57.8048       70.6994       57.4229       9.3476         10     
reco:gaushit:GausHitFinder                          1.25319       1.6456        2.24025       1.58173      0.291264        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000510042   0.000625793   0.000929628   0.000582616   0.000125029      10     
reco:reco3d:SpacePointSolver                        15.4686       25.4164       47.656        23.1132       9.65537        10     
reco:hitpdune:DisambigFromSpacePoints               1.40441       2.36243       3.73932       2.17671      0.686157        10     
reco:pandora:StandardPandora                        33.7099       53.8648       85.3525       50.6766       16.6875        10     
reco:pandoraWriter:StandardPandora                  0.42731      0.579712      0.746567      0.559362      0.0973105       10     
reco:pandoraTrack:LArPandoraTrackCreation           5.43596       8.6017        13.8658       8.52856       2.45414        10     
reco:pandoraShower:LArPandoraShowerCreation         5.80735       11.3721       21.814        9.19157       4.84541        10     
reco:pandoracalo:Calorimetry                        3.17876       4.75087       7.97215       4.67987       1.25151        10     
reco:pandoracalonosce:Calorimetry                   2.94829       4.48843       7.38918       4.42503       1.16644        10     
reco:pandorapid:Chi2ParticleID                    0.00574934    0.00711356    0.00899756    0.00676528    0.000917302      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0504091     0.0644706     0.0856288     0.0609769     0.0108737       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00586295    0.00684749    0.00828106    0.00659576    0.00064396       10     
reco:pandoraShowercalo:ShowerCalorimetry            6.94514       11.3933       22.2537       9.69386       4.3965         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       6.44979       11.0481       22.1155       9.40467       4.44557        10     
reco:emtrkmichelid:EmTrackMichelId                  197.685       256.36        335.48        254.822       39.3631        10     
reco:crttag:SingleCRTMatchingProducer              0.0122962     0.0212424     0.0332927     0.0221302    0.00697193       10     
reco:crtreco:TwoCRTMatchingProducer                0.0113706     0.0167261     0.0255228     0.0162146    0.00426449       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.972041       1.22372       1.65436       1.21615      0.200605        10     
reco:pandora2Track:LArPandoraTrackCreation          15.692        28.9649       42.4821       27.2103       9.94093        10     
reco:pandora2calo:Calorimetry                       8.42684       14.7905       21.6963       14.5706       4.66272        10     
reco:pandora2calonosce:Calorimetry                  7.82995       14.0331       21.0521       13.4686       4.51801        10     
reco:pandora2pid:Chi2ParticleID                    0.0127871     0.0172201     0.0212436     0.0177823    0.00257181       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0608453     0.0825701     0.106568      0.0817428     0.0135695       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0132143     0.0174822     0.0215443     0.0180403    0.00255957       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.79424       10.0494       17.6719       8.57452       3.8762         10     
reco:pandora2Showercalo:ShowerCalorimetry           7.1163        11.5075       18.4524       10.4117       3.59961        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.14058       9.68908       14.6541       8.99503       2.87738        10     
[art]:TriggerResults:TriggerResultInserter        3.6985e-05    5.24142e-05   0.00012005    4.6125e-05    2.30452e-05      10     
end_path:out1:RootOutput                           7.391e-06    9.5175e-06    1.7718e-05     8.133e-06    3.13352e-06      10     
end_path:out1:RootOutput(write)                     3.14722       4.04444       5.37927       3.96921      0.666618        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5349.18 MB
  Peak resident set size usage (VmHWM): 4550.19 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.15 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11450884
-rw-r--r--. 1 dune009 dune 2640843923 Jan 14 02:38 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2504246735 Jan 14 02:36 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2385600472 Jan 14 04:12 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2087531903 Jan 13 23:27 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2018428340 Jan 13 23:09 g4_stage1.root
-rw-r--r--. 1 dune009 dune   88439360 Jan 14 04:06 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243254 Jan 14 04:12 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 14 04:12 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 14 04:12 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 14 04:12 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 14 02:38 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 23:27 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 14 04:12 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 13 23:01 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 14 04:12 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 14 02:46 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 14 02:38 debugprod.log
justIN time: 2026-02-06 04:40:33 UTC       justIN version: 01.06.00