justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253908.1@dunegpschedd01.fnal.gov

Jobsub ID253908.1@dunegpschedd01.fnal.gov
Workflow ID10365
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-18 23:14:15
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-19 03:34:28
From worker nodeHostnamewn2204240.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-18 23:15:57
Input filesmonte-carlo-010365-000001
JobscriptExit code0
Real time4h (14603s)
CPU time3h (14079s = 96%)
Max RSS bytes3960823808 (3777 MiB)
Outputting started2025-11-19 03:19:20
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10365/1/001/reco_stage1.root
Finished2025-11-19 03:34:28
Saved logsjustin-logs:253908.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 100007 traces, 0 summary
Retagger: tagging trace set: gauss with 80917 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 80917 traces tagged "gauss"
FrameSaver: q=2.68465e+06 n=1459773 tag=gauss
wclsFrameSaver: saving 100007 traces tagged "wiener"
FrameSaver: q=2.80989e+06 n=1301675 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 15752 XUs and 23525 XVs -> 105 XUVs
C:0 T:1 31838 XUs and 36973 XVs -> 1802 XUVs
C:0 T:2 64550 XUs and 64967 XVs -> 2404 XUVs
C:0 T:3 36157 XUs and 24491 XVs -> 99 XUVs
C:0 T:4 11791 XUs and 12948 XVs -> 84 XUVs
C:0 T:5 40046 XUs and 32639 XVs -> 5940 XUVs
C:0 T:6 156049 XUs and 151087 XVs -> 5528 XUVs
C:0 T:7 59650 XUs and 55222 XVs -> 623 XUVs
C:0 T:8 5310 XUs and 5710 XVs -> 76 XUVs
C:0 T:9 26698 XUs and 36308 XVs -> 1933 XUVs
C:0 T:10 15204 XUs and 8594 XVs -> 1566 XUVs
C:0 T:11 48 XUs and 17 XVs -> 4 XUVs
20164 XUVs total
9120 collection wire objects
18794 potential space points
Neighbour search...
985060 tests to find 491418 neighbours
Iterating with no regularization...
Begin: 5.3204e+07
0 4.62752e+07
1 4.57815e+07
2 4.57087e+07
3 4.56886e+07
Now with regularization...
Begin: 3.81801e+07
0 3.81728e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410598525
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410598525
fLifetime: 30000 [us]
19-Nov-2025 03:19:18 GMT  Closed output file "reco_stage1.root"
19-Nov-2025 03:19:18 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          300.103       407.315       620.678       382.421       87.6225        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00211791    0.00550316     0.0115622     0.0048584    0.00251853       10     
reco:ophit:OpHitFinder                             0.248412      0.331395      0.408633      0.323108      0.0482271       10     
reco:opflash:OpFlashFinder                         0.142349      0.191696      0.260189      0.190141      0.0361269       10     
reco:opslicer:OpSlicer                             0.659274      0.823437       1.12597      0.762892      0.162502        10     
reco:caldata:DataPrepModule                         5.63058       6.63693       7.86352       6.82565      0.693727        10     
reco:wclsdatasp:WireCellToolkit                     40.5699       47.9086       56.7561       48.0876       4.37524        10     
reco:gaushit:GausHitFinder                         0.793028       1.30176       2.36969       1.26235      0.451735        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000520189   0.000780498   0.00104316    0.000769833   0.000158255      10     
reco:reco3d:SpacePointSolver                        12.5624       16.6492       27.0658       16.1943       3.99862        10     
reco:hitpdune:DisambigFromSpacePoints               1.11315       1.74349       2.50814       1.66714      0.440451        10     
reco:pandora:StandardPandora                        21.7616       30.6557       55.5795       27.2213       9.87745        10     
reco:pandoraWriter:StandardPandora                 0.446421      0.562686      0.685636      0.549922      0.0751978       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.49726       7.23664       12.6097       6.99044       2.43818        10     
reco:pandoraShower:LArPandoraShowerCreation         2.85715       7.66265       14.635        6.18495       3.77858        10     
reco:pandoracalo:Calorimetry                        1.75412       3.65633       5.33508       3.60137       1.0869         10     
reco:pandoracalonosce:Calorimetry                   1.65567       3.11105       5.89439       2.92102       1.17258        10     
reco:pandorapid:Chi2ParticleID                    0.00401014     0.0065148    0.00900502    0.00619768     0.0015229       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.040843      0.0531971     0.0782403     0.0482575     0.0131436       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00367682    0.00575949    0.00871348    0.00545482    0.00141143       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.63551       7.1734        17.3171       5.46863       4.49788        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.07865       7.35903       15.5455       5.48167       4.00503        10     
reco:emtrkmichelid:EmTrackMichelId                  158.794       200.391       296.27        193.415       36.1878        10     
reco:crttag:SingleCRTMatchingProducer              0.0159864     0.0300303     0.048918      0.0239748     0.0121608       10     
reco:crtreco:TwoCRTMatchingProducer               0.00953705     0.0238993     0.0534555     0.022166      0.0127999       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.554951      0.927078       1.23208      0.925022      0.211274        10     
reco:pandora2Track:LArPandoraTrackCreation          8.00646       21.3275       35.8925       20.4638       9.17828        10     
reco:pandora2calo:Calorimetry                       3.57682       9.38943       15.8534       10.3193       3.81455        10     
reco:pandora2calonosce:Calorimetry                  3.15205       8.34766       15.3851       8.7456        3.41117        10     
reco:pandora2pid:Chi2ParticleID                   0.00665269     0.0152962     0.0221428     0.0164525    0.00508451       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0391801     0.0743548     0.107657      0.0720218     0.0217522       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00824398     0.013888      0.0198399     0.0139694    0.00409714       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.1291        6.45423       11.729        6.55709       2.97213        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.84694       7.52335       14.5584       8.06435       3.32012        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.72603       6.53305       12.8414       5.3667        3.39594        10     
[art]:TriggerResults:TriggerResultInserter        6.8909e-05    9.06565e-05   0.000189496   8.0637e-05    3.39438e-05      10     
end_path:out1:RootOutput                           7.955e-06    1.31187e-05   2.2102e-05     1.323e-05    3.9633e-06       10     
end_path:out1:RootOutput(write)                     2.20205       2.94904       3.82549       2.84642      0.509561        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4769.59 MB
  Peak resident set size usage (VmHWM): 3960.82 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.96 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9403572
-rw-r--r-- 1 dune01 dune01 2226752115 Nov 19 02:09 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2111310734 Nov 19 02:06 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1876835088 Nov 19 03:19 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1712100506 Nov 18 23:39 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1635927337 Nov 18 23:24 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   65702030 Nov 19 03:14 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242913 Nov 19 03:19 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 19 03:19 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 19 03:19 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 19 03:19 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 19 02:09 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 23:39 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 19 03:19 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 18 23:15 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 19 03:19 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 19 02:14 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 19 02:09 debugprod.log
justIN time: 2026-02-09 05:01:07 UTC       justIN version: 01.06.00