justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271062.0@dunegpschedd02.fnal.gov

Jobsub ID271062.0@dunegpschedd02.fnal.gov
Workflow ID11790
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 06:13:01
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2026-01-13 11:33:41
From worker nodeHostnamewn-la-21.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 06:14:10
Input filesmonte-carlo-011790-000001
JobscriptExit code0
Real time5h (19048s)
CPU time5h (18844s = 98%)
Max RSS bytes4446240768 (4240 MiB)
Outputting started2026-01-13 11:31:39
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11790/1/001/reco_stage1.root
Finished2026-01-13 11:33:41
Saved logsjustin-logs:271062.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 186947 traces, 0 summary
Retagger: tagging trace set: gauss with 143290 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 143290 traces tagged "gauss"
FrameSaver: q=5.08401e+06 n=2595589 tag=gauss
wclsFrameSaver: saving 186947 traces tagged "wiener"
FrameSaver: q=5.37062e+06 n=2307639 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 47908 XUs and 53550 XVs -> 501 XUVs
C:0 T:1 107631 XUs and 112823 XVs -> 7449 XUVs
C:0 T:2 142844 XUs and 157859 XVs -> 12204 XUVs
C:0 T:3 61053 XUs and 66801 XVs -> 578 XUVs
C:0 T:4 16851 XUs and 15270 XVs -> 251 XUVs
C:0 T:5 67120 XUs and 49509 XVs -> 11378 XUVs
C:0 T:6 153954 XUs and 187247 XVs -> 7298 XUVs
C:0 T:7 61367 XUs and 70542 XVs -> 989 XUVs
C:0 T:8 15405 XUs and 13533 XVs -> 270 XUVs
C:0 T:9 215393 XUs and 163079 XVs -> 38899 XUVs
C:0 T:10 74269 XUs and 57183 XVs -> 9072 XUVs
C:0 T:11 281 XUs and 214 XVs -> 23 XUVs
88912 XUVs total
16853 collection wire objects
81418 potential space points
Neighbour search...
19327822 tests to find 7726136 neighbours
Iterating with no regularization...
Begin: 9.38015e+07
0 7.56515e+07
1 7.4245e+07
2 7.4061e+07
3 7.40082e+07
Now with regularization...
Begin: 5.97765e+07
0 5.9714e+07
1 5.96686e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411710787
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411710787
fLifetime: 30000 [us]
13-Jan-2026 12:31:37 CET  Closed output file "reco_stage1.root"
13-Jan-2026 12:31:37 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          422.18        544.089       778.59        511.235       112.562        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00112589    0.00612165     0.0110821    0.00640372    0.00302443       10     
reco:ophit:OpHitFinder                             0.381198      0.454133      0.544315      0.441935      0.0595183       10     
reco:opflash:OpFlashFinder                         0.233282      0.343238      0.475284      0.328646      0.0774554       10     
reco:opslicer:OpSlicer                             0.672228      0.895541       1.15887      0.855429      0.183854        10     
reco:caldata:DataPrepModule                         6.33206       6.5734        7.11605       6.51356       0.21244        10     
reco:wclsdatasp:WireCellToolkit                     46.5324       61.5291       71.3801       63.8879       8.11004        10     
reco:gaushit:GausHitFinder                          1.13927       1.56673       2.1913        1.48829      0.286492        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000467127   0.000616752   0.000817741   0.000632675   0.000112285      10     
reco:reco3d:SpacePointSolver                        16.1756       30.5269       52.3905       26.2038       12.0918        10     
reco:hitpdune:DisambigFromSpacePoints               1.59863       2.4119        3.95675       2.1236       0.783683        10     
reco:pandora:StandardPandora                        30.9499       57.865        108.684       49.5794       25.2998        10     
reco:pandoraWriter:StandardPandora                 0.456051      0.585453      0.724863       0.56446      0.0848448       10     
reco:pandoraTrack:LArPandoraTrackCreation           6.04705       9.46965       12.9474       9.71761       2.13534        10     
reco:pandoraShower:LArPandoraShowerCreation         7.05089       11.5918       20.3084       10.1556       4.2848         10     
reco:pandoracalo:Calorimetry                        3.22066       5.05888       7.68859       5.1128        1.19707        10     
reco:pandoracalonosce:Calorimetry                   3.05707       4.89328       7.24912       4.94258       1.12911        10     
reco:pandorapid:Chi2ParticleID                    0.00549027    0.00721259    0.00927337    0.00729619    0.00115164       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0508814     0.0616172     0.0777889     0.0582365    0.00791841       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00559348    0.00706537    0.00867691    0.00724672    0.000882189      10     
reco:pandoraShowercalo:ShowerCalorimetry            7.61225       11.5975       20.2061       10.0614       3.98006        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        6.72         11.2002       20.042        9.30028       4.13668        10     
reco:emtrkmichelid:EmTrackMichelId                  184.654       234.607       307.254       225.903       35.9567        10     
reco:crttag:SingleCRTMatchingProducer              0.0142347     0.0210901     0.0343063     0.0199112    0.00642068       10     
reco:crtreco:TwoCRTMatchingProducer                0.0122611     0.017715      0.0302491     0.0152396    0.00530273       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.941249       1.21251       1.51613       1.12404      0.198303        10     
reco:pandora2Track:LArPandoraTrackCreation          17.2315       30.1927       52.7351       26.5726       11.6397        10     
reco:pandora2calo:Calorimetry                       8.66281       14.0373       21.5517       13.1788       4.42683        10     
reco:pandora2calonosce:Calorimetry                  7.98769       13.0352       19.7085       12.3711       4.06394        10     
reco:pandora2pid:Chi2ParticleID                    0.0135043     0.0177439     0.0225217     0.0178282    0.00269695       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0669678     0.0890468     0.189729      0.0763635     0.0347327       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0133099     0.017298      0.0225384     0.016735     0.00277407       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.07548       9.6792        14.5554       8.42209       2.82871        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.58256       10.9542       15.4198       10.3143       2.59407        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.9018        9.31409       12.7748       8.61624       1.99142        10     
[art]:TriggerResults:TriggerResultInserter        4.0461e-05    5.62033e-05   0.00014059    4.6901e-05    2.84635e-05      10     
end_path:out1:RootOutput                           7.352e-06    8.7314e-06    1.4302e-05    8.1025e-06    1.9154e-06       10     
end_path:out1:RootOutput(write)                     3.12341       3.99577       5.21005       3.7601       0.642221        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5248.78 MB
  Peak resident set size usage (VmHWM): 4446.24 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.11 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11702720
-rw-r--r--. 1 dune009 dune 2689865830 Jan 13 10:58 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2554783738 Jan 13 10:55 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2427256491 Jan 13 12:31 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2157696553 Jan 13 07:41 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2065696902 Jan 13 07:22 g4_stage1.root
-rw-r--r--. 1 dune009 dune   87668040 Jan 13 12:22 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243078 Jan 13 12:31 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 13 12:31 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 13 12:31 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 13 12:31 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 13 10:58 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 07:41 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 12:31 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 13 07:14 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 13 12:31 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 13 11:06 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 13 10:58 debugprod.log
justIN time: 2026-02-06 07:40:10 UTC       justIN version: 01.06.00