justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296535.1@dunegpschedd02.fnal.gov

Jobsub ID296535.1@dunegpschedd02.fnal.gov
Workflow ID12529
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-29 16:40:09
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2026-01-29 20:30:29
From worker nodeHostnamehnode51.pic.es
cpuinfoAMD EPYC 7402P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-29 16:40:49
Input filesmonte-carlo-012529-000001
JobscriptExit code0
Real time3h (13698s)
CPU time3h (12109s = 88%)
Max RSS bytes4220792832 (4025 MiB)
Outputting started2026-01-29 20:29:08
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12529/1/001/reco_stage1.root
Finished2026-01-29 20:30:29
Saved logsjustin-logs:296535.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

repare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 149438 traces, 0 summary
Retagger: tagging trace set: gauss with 110147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 110147 traces tagged "gauss"
FrameSaver: q=3.86533e+06 n=1900414 tag=gauss
wclsFrameSaver: saving 149438 traces tagged "wiener"
FrameSaver: q=4.12236e+06 n=1665963 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 25304 XUs and 38544 XVs -> 293 XUVs
C:0 T:1 54600 XUs and 85669 XVs -> 6878 XUVs
C:0 T:2 104851 XUs and 144127 XVs -> 6698 XUVs
C:0 T:3 52156 XUs and 59526 XVs -> 359 XUVs
C:0 T:4 12804 XUs and 20549 XVs -> 226 XUVs
C:0 T:5 75072 XUs and 126939 XVs -> 10628 XUVs
C:0 T:6 125682 XUs and 166973 XVs -> 19007 XUVs
C:0 T:7 40969 XUs and 43410 XVs -> 709 XUVs
C:0 T:8 6431 XUs and 9073 XVs -> 163 XUVs
C:0 T:9 54623 XUs and 127119 XVs -> 11192 XUVs
C:0 T:10 46533 XUs and 23796 XVs -> 5013 XUVs
C:0 T:11 71 XUs and 60 XVs -> 10 XUVs
61176 XUVs total
13917 collection wire objects
56214 potential space points
Neighbour search...
9703768 tests to find 4303302 neighbours
Iterating with no regularization...
Begin: 7.25372e+07
0 5.60118e+07
1 5.48205e+07
2 5.46053e+07
3 5.45271e+07
4 5.44871e+07
Now with regularization...
Begin: 4.27933e+07
0 4.27098e+07
1 4.26542e+07
2 4.26114e+07
3 4.25759e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 412041387
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412041387
fLifetime: 30000 [us]
29-Jan-2026 21:29:06 CET  Closed output file "reco_stage1.root"
29-Jan-2026 21:29:06 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          279.014       407.752       571.066        414.1        98.9527        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00123481    0.00312243    0.00398087    0.00348138    0.000900217      10     
reco:ophit:OpHitFinder                             0.224043      0.301615      0.430876      0.294321      0.0511027       10     
reco:opflash:OpFlashFinder                         0.119007      0.156819      0.257321      0.148386      0.0371272       10     
reco:opslicer:OpSlicer                              0.48827      0.731004       1.10474      0.708894       0.17163        10     
reco:caldata:DataPrepModule                         4.38143       5.81308       7.2557        5.93451      0.903025        10     
reco:wclsdatasp:WireCellToolkit                     30.5059       37.0516       40.713        38.693        3.82628        10     
reco:gaushit:GausHitFinder                         0.844547       1.26138       2.06748       1.16498      0.375466        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000431714   0.000631612   0.00101434    0.000584773   0.000165483      10     
reco:reco3d:SpacePointSolver                        5.32258       14.7945        35.11        13.2492       8.31125        10     
reco:hitpdune:DisambigFromSpacePoints              0.668985       1.57997       2.79616       1.60055      0.666453        10     
reco:pandora:StandardPandora                        17.926        33.7306       55.3466       34.2096       11.9968        10     
reco:pandoraWriter:StandardPandora                  0.29716      0.423941      0.516268       0.42526      0.0647903       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.34272       4.15497       9.01456       3.51553       1.86486        10     
reco:pandoraShower:LArPandoraShowerCreation         2.18333       4.60792       11.2384       3.72962       2.50512        10     
reco:pandoracalo:Calorimetry                        1.1421        2.32952       5.89461       2.02895       1.31435        10     
reco:pandoracalonosce:Calorimetry                   1.0007        2.21243       5.81746       1.86904       1.32717        10     
reco:pandorapid:Chi2ParticleID                    0.00413907    0.00606346    0.00895077    0.00542245    0.00166204       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0346477     0.0554474     0.0763679     0.0558152     0.0143941       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00417921    0.00609767    0.00915102    0.00626831     0.0016083       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.91447       4.32775       9.16558       3.99155       2.17802        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.79462       4.3964        11.8142       3.60136       2.9042         10     
reco:emtrkmichelid:EmTrackMichelId                  173.34        244.158       325.469       242.298       52.8646        10     
reco:crttag:SingleCRTMatchingProducer             0.00691095     0.0153836     0.0243468     0.0149663    0.00472682       10     
reco:crtreco:TwoCRTMatchingProducer               0.00571691     0.0109772     0.0154426     0.0105088    0.00293215       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.650706       1.03252       1.29922       1.03703      0.193467        10     
reco:pandora2Track:LArPandoraTrackCreation          5.5178        14.6056       22.6884       15.1243       5.23221        10     
reco:pandora2calo:Calorimetry                       2.20245       6.06383       8.9335        5.88995       2.14416        10     
reco:pandora2calonosce:Calorimetry                  2.32592       5.81241       9.67134       5.59256       2.37854        10     
reco:pandora2pid:Chi2ParticleID                   0.00835876     0.0163196     0.022948      0.0171197    0.00442823       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0380703     0.076335      0.119864      0.0756232     0.0239212       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0083231     0.0158207     0.0211718     0.0165302    0.00382497       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.96072       5.11155       8.50925       5.37863       1.90557        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.03488       5.15791       7.87083       5.40641       1.93246        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.72989       4.6713        8.59043       4.74824       2.0654         10     
[art]:TriggerResults:TriggerResultInserter        2.3181e-05    4.14063e-05   0.000169215   2.5871e-05    4.27483e-05      10     
end_path:out1:RootOutput                           4.79e-06     4.54963e-05   0.000398192   5.4055e-06    0.00011759       10     
end_path:out1:RootOutput(write)                     1.86061       2.91528       4.37251       2.94324      0.646067        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5194.14 MB
  Peak resident set size usage (VmHWM): 4220.79 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.70 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9772436
-rw-r--r-- 1 dune001 dune 2307443557 Jan 29 20:18 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2198235259 Jan 29 20:15 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 1968930375 Jan 29 21:29 reco_stage1.root
-rw-r--r-- 1 dune001 dune 1762636117 Jan 29 18:09 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1699660907 Jan 29 17:50 g4_stage1.root
-rw-r--r-- 1 dune001 dune   69437300 Jan 29 21:22 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     243930 Jan 29 21:29 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Jan 29 21:29 mem.db
-rw-r--r-- 1 dune001 dune      62376 Jan 29 21:29 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Jan 29 21:29 time.db
-rw-r--r-- 1 dune001 dune        519 Jan 29 20:18 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 29 18:09 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 29 21:29 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Jan 29 17:40 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Jan 29 21:29 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Jan 29 20:27 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Jan 29 20:18 debugprod.log
justIN time: 2026-02-04 15:21:16 UTC       justIN version: 01.06.00