justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 277471.1@dunegpschedd01.fnal.gov

Jobsub ID277471.1@dunegpschedd01.fnal.gov
Workflow ID11861
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 21:59:53
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-14 03:42:35
From worker nodeHostnamewn-la-10.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 22:01:27
Input filesmonte-carlo-011861-000001
JobscriptExit code0
Real time5h (20363s)
CPU time5h (20194s = 99%)
Max RSS bytes4843765760 (4619 MiB)
Outputting started2026-01-14 03:40:51
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11861/1/001/reco_stage1.root
Finished2026-01-14 03:42:35
Saved logsjustin-logs:277471.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ol adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 126391 traces, 0 summary
Retagger: tagging trace set: gauss with 98784 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 98784 traces tagged "gauss"
FrameSaver: q=3.62144e+06 n=1791356 tag=gauss
wclsFrameSaver: saving 126391 traces tagged "wiener"
FrameSaver: q=3.82765e+06 n=1610658 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 42682 XUs and 50128 XVs -> 905 XUVs
C:0 T:1 79190 XUs and 88480 XVs -> 7123 XUVs
C:0 T:2 115869 XUs and 106812 XVs -> 4939 XUVs
C:0 T:3 59356 XUs and 43656 XVs -> 353 XUVs
C:0 T:4 12947 XUs and 12370 XVs -> 217 XUVs
C:0 T:5 47560 XUs and 41656 XVs -> 7536 XUVs
C:0 T:6 100757 XUs and 84889 XVs -> 3178 XUVs
C:0 T:7 45154 XUs and 35528 XVs -> 287 XUVs
C:0 T:8 5242 XUs and 5216 XVs -> 32 XUVs
C:0 T:9 24499 XUs and 31182 XVs -> 1971 XUVs
C:0 T:10 48738 XUs and 32604 XVs -> 6619 XUVs
C:0 T:11 176 XUs and 119 XVs -> 20 XUVs
33180 XUVs total
10102 collection wire objects
30629 potential space points
Neighbour search...
1890465 tests to find 965018 neighbours
Iterating with no regularization...
Begin: 8.19128e+07
0 7.05112e+07
1 6.94864e+07
2 6.93348e+07
3 6.92898e+07
Now with regularization...
Begin: 5.84493e+07
0 5.84146e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411723965
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411723965
fLifetime: 30000 [us]
14-Jan-2026 04:40:49 CET  Closed output file "reco_stage1.root"
14-Jan-2026 04:40:49 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          468.729       612.602       764.599       583.857       103.361        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00138918    0.00607943     0.0107624    0.00601789    0.00301573       10     
reco:ophit:OpHitFinder                             0.398681      0.497158       0.60174      0.495327      0.050137        10     
reco:opflash:OpFlashFinder                         0.246707      0.395877      0.492823      0.373765      0.0702487       10     
reco:opslicer:OpSlicer                             0.817855       1.01858       1.25192       1.01977      0.132638        10     
reco:caldata:DataPrepModule                         6.38081       6.57428       7.24812       6.52072      0.233705        10     
reco:wclsdatasp:WireCellToolkit                     46.2876       66.7493       73.9116       71.2357       9.81316        10     
reco:gaushit:GausHitFinder                          1.43837       1.74678       2.14699       1.6244       0.236338        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000445192   0.00058876    0.000928422   0.000544475   0.000128468      10     
reco:reco3d:SpacePointSolver                        19.8833        32.31         57.15        29.8095       11.0871        10     
reco:hitpdune:DisambigFromSpacePoints               2.01694       2.85516       3.98592       2.74533      0.646423        10     
reco:pandora:StandardPandora                        45.3801       77.0185       120.932       67.2518       28.1035        10     
reco:pandoraWriter:StandardPandora                 0.489598      0.618592      0.750129      0.597788      0.0878098       10     
reco:pandoraTrack:LArPandoraTrackCreation           6.13101       9.17311       14.1845       8.37336       2.40641        10     
reco:pandoraShower:LArPandoraShowerCreation         6.51971       12.4475       24.7273       11.0562       5.31036        10     
reco:pandoracalo:Calorimetry                        3.62453       5.09644       8.45719       4.73865       1.32675        10     
reco:pandoracalonosce:Calorimetry                   3.42571       4.93866       8.02363       4.45077       1.38708        10     
reco:pandorapid:Chi2ParticleID                     0.0063893    0.00769097      0.01083     0.00730156    0.00117783       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0561691     0.0699488     0.0856171     0.0677194     0.0102967       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00623102    0.00734686    0.00854493     0.0072222    0.000748117      10     
reco:pandoraShowercalo:ShowerCalorimetry            6.72994       13.261        22.5659       11.1899       5.63912        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       6.59634       12.7016       20.6798       10.9839       5.14959        10     
reco:emtrkmichelid:EmTrackMichelId                  221.155       269.942       321.114       261.373       33.9631        10     
reco:crttag:SingleCRTMatchingProducer              0.0143347     0.0220907     0.0298759     0.0232827    0.00492021       10     
reco:crtreco:TwoCRTMatchingProducer                0.0124409     0.0163571     0.0204466     0.0165729    0.00258941       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.01362       1.33132       1.58256       1.31958      0.168844        10     
reco:pandora2Track:LArPandoraTrackCreation          18.8189       29.9041       42.9555       28.3262       8.20345        10     
reco:pandora2calo:Calorimetry                       9.86603       14.8703       20.3159       14.6886       3.44665        10     
reco:pandora2calonosce:Calorimetry                  9.27023       13.8596       18.6944       13.6736       3.15692        10     
reco:pandora2pid:Chi2ParticleID                    0.0156523     0.0187813     0.0223156     0.0185116    0.00245623       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0739863     0.0889745     0.107259      0.0861445     0.0117223       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0155684     0.0186776     0.0222972     0.0182724    0.00243713       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.10982       9.57696       12.7431       9.14961       2.25461        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.34142       11.1747       13.9646       11.4917       2.17671        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.1438         9.586        11.7624       9.88064       1.89629        10     
[art]:TriggerResults:TriggerResultInserter        3.3771e-05    4.73394e-05   0.000119829   3.9243e-05    2.43597e-05      10     
end_path:out1:RootOutput                           7.08e-06     8.2051e-06     1.335e-05    7.5965e-06    1.74848e-06      10     
end_path:out1:RootOutput(write)                     3.59215       4.45596       5.41303       4.29183      0.594975        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5649.89 MB
  Peak resident set size usage (VmHWM): 4843.77 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.14 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 12490352
-rw-r--r--. 1 dune009 dune 2852364824 Jan 14 02:56 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2705538284 Jan 14 02:53 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2627034530 Jan 14 04:40 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2304426028 Jan 13 23:29 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2203743287 Jan 13 23:10 g4_stage1.root
-rw-r--r--. 1 dune009 dune   96396980 Jan 14 04:35 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243198 Jan 14 04:40 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 14 04:40 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 14 04:40 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 14 04:40 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 14 02:56 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 23:29 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 14 04:40 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 13 23:01 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 14 04:40 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 14 03:07 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 14 02:56 debugprod.log
justIN time: 2026-02-06 04:40:09 UTC       justIN version: 01.06.00