justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 247057.1@dunegpschedd02.fnal.gov

Jobsub ID247057.1@dunegpschedd02.fnal.gov
Workflow ID10252
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-16 10:29:47
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-16 14:25:38
From worker nodeHostnamewn-lb-05.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-16 10:30:48
Input filesmonte-carlo-010252-000001
JobscriptExit code0
Real time3h (13696s)
CPU time3h (13571s = 99%)
Max RSS bytes4099760128 (3909 MiB)
Outputting started2025-11-16 14:19:05
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10252/1/001/reco_stage1.root
Finished2025-11-16 14:25:38
Saved logsjustin-logs:247057.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

lSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 95494 traces, 0 summary
Retagger: tagging trace set: gauss with 72572 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72572 traces tagged "gauss"
FrameSaver: q=2.40669e+06 n=1262339 tag=gauss
wclsFrameSaver: saving 95494 traces tagged "wiener"
FrameSaver: q=2.566e+06 n=1109457 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 22719 XUs and 21569 XVs -> 138 XUVs
C:0 T:1 43279 XUs and 42428 XVs -> 2652 XUVs
C:0 T:2 90028 XUs and 97623 XVs -> 4479 XUVs
C:0 T:3 43124 XUs and 40163 XVs -> 253 XUVs
C:0 T:4 8053 XUs and 9722 XVs -> 77 XUVs
C:0 T:5 35245 XUs and 32148 XVs -> 4421 XUVs
C:0 T:6 81648 XUs and 157102 XVs -> 12643 XUVs
C:0 T:7 27747 XUs and 39398 XVs -> 476 XUVs
C:0 T:8 3642 XUs and 5822 XVs -> 75 XUVs
C:0 T:9 31497 XUs and 68174 XVs -> 5091 XUVs
C:0 T:10 11592 XUs and 7768 XVs -> 1555 XUVs
C:0 T:11 26 XUs and 25 XVs -> 0 XUVs
31860 XUVs total
9420 collection wire objects
29390 potential space points
Neighbour search...
4668668 tests to find 2037948 neighbours
Iterating with no regularization...
Begin: 5.62291e+07
0 4.18579e+07
1 4.10032e+07
2 4.08708e+07
3 4.08302e+07
Now with regularization...
Begin: 3.29845e+07
0 3.28897e+07
1 3.28413e+07
2 3.28088e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410547379
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410547379
fLifetime: 30000 [us]
16-Nov-2025 15:19:03 CET  Closed output file "reco_stage1.root"
16-Nov-2025 15:19:03 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          272.262       344.114       431.312       316.147       66.1395        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00133082    0.00564392    0.00966258     0.0058019    0.00240383       10     
reco:ophit:OpHitFinder                             0.243714      0.294103      0.425155      0.270317      0.0582294       10     
reco:opflash:OpFlashFinder                         0.127831       0.17185      0.242775      0.160869      0.0385093       10     
reco:opslicer:OpSlicer                             0.476812      0.608874      0.898976      0.565994      0.135086        10     
reco:caldata:DataPrepModule                         4.69381       5.18161       6.95216       4.8354       0.768224        10     
reco:wclsdatasp:WireCellToolkit                     34.9235       45.0803       49.7071       45.6706       3.82225        10     
reco:gaushit:GausHitFinder                         0.756554       1.01926       1.25968       1.04628      0.156717        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000377629   0.000762635   0.00117446    0.000726658   0.000230849      10     
reco:reco3d:SpacePointSolver                        7.46004       13.439        25.1384       10.7356       5.32212        10     
reco:hitpdune:DisambigFromSpacePoints               0.6966         1.15         1.68933       1.04783      0.302315        10     
reco:pandora:StandardPandora                        16.4444       29.7308       53.0606       26.8466       11.2961        10     
reco:pandoraWriter:StandardPandora                 0.341762       0.4408       0.543102      0.441602      0.054524        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.36737       7.78145       12.4517       6.48764       3.06157        10     
reco:pandoraShower:LArPandoraShowerCreation         5.26394       8.7579        13.8604       7.34781       2.86233        10     
reco:pandoracalo:Calorimetry                        2.01091       3.89917       7.24912       2.86797       1.89035        10     
reco:pandoracalonosce:Calorimetry                   1.84566       3.66366       6.96472       2.40081       1.83241        10     
reco:pandorapid:Chi2ParticleID                    0.00329185    0.00505818    0.00827119    0.00413099    0.00167627       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0327011     0.0418894     0.0581155     0.039456     0.00811733       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00363072    0.00497965    0.00756431    0.00438348    0.00124008       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.64171       8.01243       13.2573       5.94224       3.35464        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.8986        7.47008       14.407        5.21618       3.62169        10     
reco:emtrkmichelid:EmTrackMichelId                  117.989       144.583       185.251       139.969       21.3783        10     
reco:crttag:SingleCRTMatchingProducer              0.0110985     0.0206356     0.0362627     0.0186168    0.00723154       10     
reco:crtreco:TwoCRTMatchingProducer               0.00729447     0.0167522     0.0270223     0.0140397     0.0067663       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.475849      0.691268      0.917767       0.71639      0.126225        10     
reco:pandora2Track:LArPandoraTrackCreation          10.8393       20.3149       33.9363       20.1245       6.57104        10     
reco:pandora2calo:Calorimetry                       5.12298       9.20667       14.9287       8.70107       3.5165         10     
reco:pandora2calonosce:Calorimetry                  4.80822       8.37466       14.7563       7.1844        3.33623        10     
reco:pandora2pid:Chi2ParticleID                   0.00764057    0.00964953     0.012416     0.00909548    0.00143224       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0412725     0.048294      0.0553662     0.047999     0.00452527       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00781789    0.00971967     0.0125189    0.00916388    0.00148805       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.49734       8.14113       12.6237       7.87896       2.13791        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.86546       7.21949       11.8126       6.19841       2.62674        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.2271        6.04274       10.7013       5.30812       2.06121        10     
[art]:TriggerResults:TriggerResultInserter        3.7387e-05    6.86107e-05   0.000167844   5.7407e-05    3.61539e-05      10     
end_path:out1:RootOutput                           7.391e-06    1.01201e-05    2.012e-05     8.513e-06    3.88918e-06      10     
end_path:out1:RootOutput(write)                     1.92994       2.38668       3.55538       2.25226      0.468113        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4904 MB
  Peak resident set size usage (VmHWM): 4099.76 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.15 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9356136
-rw-r--r--. 1 dune009 dune 2223176005 Nov 16 14:19 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2114651033 Nov 16 14:17 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 1869047796 Nov 16 15:19 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1686454809 Nov 16 11:49 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1621616030 Nov 16 11:36 g4_stage1.root
-rw-r--r--. 1 dune009 dune   65118480 Nov 16 15:15 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242787 Nov 16 15:19 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 16 15:19 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 16 15:19 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 16 15:19 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 16 14:19 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 16 11:49 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 16 15:19 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 16 11:30 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 16 15:19 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 16 14:26 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 16 14:20 debugprod.log
justIN time: 2026-02-09 10:52:26 UTC       justIN version: 01.06.00