justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 244865.1@dunegpschedd02.fnal.gov

Jobsub ID244865.1@dunegpschedd02.fnal.gov
Workflow ID10192
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-14 18:39:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-14 20:42:24
From worker nodeHostnamedunegli-7598443-0-fnpc22028.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-14 18:40:14
Input filesmonte-carlo-010192-000001
JobscriptExit code0
Real time2h (7311s)
CPU time2h (7224s = 98%)
Max RSS bytes4093874176 (3904 MiB)
Outputting started2025-11-14 20:42:06
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10192/1/001/reco_stage1.root
Finished2025-11-14 20:42:24
Saved logsjustin-logs:244865.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

ScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 121762 traces, 0 summary
Retagger: tagging trace set: gauss with 92115 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 92115 traces tagged "gauss"
FrameSaver: q=3.31795e+06 n=1624126 tag=gauss
wclsFrameSaver: saving 121762 traces tagged "wiener"
FrameSaver: q=3.52561e+06 n=1438349 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33315 XUs and 35130 XVs -> 362 XUVs
C:0 T:1 86401 XUs and 64402 XVs -> 7906 XUVs
C:0 T:2 170947 XUs and 111775 XVs -> 13849 XUVs
C:0 T:3 51989 XUs and 41625 XVs -> 542 XUVs
C:0 T:4 7488 XUs and 5857 XVs -> 115 XUVs
C:0 T:5 28074 XUs and 18146 XVs -> 3920 XUVs
C:0 T:6 178841 XUs and 121857 XVs -> 14803 XUVs
C:0 T:7 44329 XUs and 39219 XVs -> 524 XUVs
C:0 T:8 10400 XUs and 10638 XVs -> 501 XUVs
C:0 T:9 97505 XUs and 93441 XVs -> 12605 XUVs
C:0 T:10 30491 XUs and 9495 XVs -> 2982 XUVs
C:0 T:11 161 XUs and 64 XVs -> 23 XUVs
58132 XUVs total
11319 collection wire objects
51389 potential space points
Neighbour search...
9096011 tests to find 4048152 neighbours
Iterating with no regularization...
Begin: 6.5606e+07
0 5.40162e+07
1 5.31429e+07
2 5.29896e+07
3 5.29369e+07
Now with regularization...
Begin: 4.28433e+07
0 4.27925e+07
1 4.27439e+07
2 4.27071e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410512001
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410512001
fLifetime: 30000 [us]
14-Nov-2025 20:42:04 UTC  Closed output file "reco_stage1.root"
14-Nov-2025 20:42:04 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          166.512       218.945       268.124       218.716       30.0259        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000678741   0.00561474     0.0214055    0.00346226    0.00590034       10     
reco:ophit:OpHitFinder                             0.192663      0.234197      0.295525      0.235375      0.0276173       10     
reco:opflash:OpFlashFinder                         0.107455      0.125983      0.146577       0.12491      0.0117672       10     
reco:opslicer:OpSlicer                             0.354712       0.42767      0.501672      0.427714      0.0385151       10     
reco:caldata:DataPrepModule                         3.49915       3.7698        4.23449       3.7385       0.181012        10     
reco:wclsdatasp:WireCellToolkit                     22.1613       24.7102       27.3199       24.8679       1.45671        10     
reco:gaushit:GausHitFinder                         0.585382      0.767579      0.938169      0.751282      0.112253        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000316097   0.000393865   0.000525602   0.000400025    5.542e-05       10     
reco:reco3d:SpacePointSolver                        5.89073       11.4539       16.7978       12.0122       3.78641        10     
reco:hitpdune:DisambigFromSpacePoints               0.77726       1.07533       1.44127       1.06568      0.222415        10     
reco:pandora:StandardPandora                        11.4357       19.797        32.5397       18.9265       5.67289        10     
reco:pandoraWriter:StandardPandora                 0.227895      0.322198      0.466388      0.334097      0.0750383       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.52097       2.2592        3.00755       2.34246      0.443804        10     
reco:pandoraShower:LArPandoraShowerCreation         1.63912       2.49377       3.13113       2.46098      0.553903        10     
reco:pandoracalo:Calorimetry                       0.872032       1.38908       1.87443       1.44451      0.277199        10     
reco:pandoracalonosce:Calorimetry                  0.795232       1.24481       1.60103       1.28389      0.230131        10     
reco:pandorapid:Chi2ParticleID                    0.00300564    0.00356655    0.00434501    0.00357013    0.000395812      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0264419     0.0325632     0.0381024     0.0327208    0.00346295       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00282036    0.00332791    0.00388668    0.00334746    0.00028048       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.6722        2.79359       3.58148       2.80318      0.663814        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.59852       2.70734       3.43102       2.74217      0.637014        10     
reco:emtrkmichelid:EmTrackMichelId                  95.5274       115.919       138.865       116.493       13.166         10     
reco:crttag:SingleCRTMatchingProducer             0.00791523    0.00989624     0.0119766    0.00972846    0.00146152       10     
reco:crtreco:TwoCRTMatchingProducer               0.00459152    0.00572144    0.00733309    0.00567136    0.000728405      10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.540267      0.675941      0.926686      0.646635       0.10964        10     
reco:pandora2Track:LArPandoraTrackCreation          4.52995       7.3362        9.19582       7.2678        1.31107        10     
reco:pandora2calo:Calorimetry                       2.3268        3.99777       5.06476       3.83717      0.759899        10     
reco:pandora2calonosce:Calorimetry                  2.24906       3.7665        4.88569       3.63119      0.774652        10     
reco:pandora2pid:Chi2ParticleID                   0.00656501    0.00805341    0.00964859    0.00815936    0.00096028       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0313491     0.0414762     0.0512455     0.0409462    0.00562809       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00641074    0.00779455    0.00885848    0.00780004    0.000780543      10     
reco:pandora2Shower:LArPandoraShowerCreation        1.81479       2.72779       3.35474       2.69565      0.456328        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.04026       3.49047       4.41514       3.42852       0.68481        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.86328       3.18209       4.03069       3.20334      0.664076        10     
[art]:TriggerResults:TriggerResultInserter        3.4555e-05    5.0394e-05    9.2124e-05    4.42535e-05   1.61921e-05      10     
end_path:out1:RootOutput                           1.013e-05    1.28675e-05   1.5609e-05    1.24085e-05   1.9288e-06       10     
end_path:out1:RootOutput(write)                     1.62012       2.08091       2.5572        2.01332      0.315178        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4906.73 MB
  Peak resident set size usage (VmHWM): 4093.87 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.23 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10020736
-rw-r--r-- 1 dunegli fnalgrid 2361196917 Nov 14 20:04 detsim_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2243796119 Nov 14 20:02 detsim_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2055564710 Nov 14 20:42 reco_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 1799606457 Nov 14 18:50 g4_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 1723699558 Nov 14 18:43 g4_stage1.root
-rw-r--r-- 1 dunegli fnalgrid   76738000 Nov 14 20:39 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid     243088 Nov 14 20:42 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid     217088 Nov 14 20:42 mem.db
-rw-r--r-- 1 dunegli fnalgrid      62376 Nov 14 20:42 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid      36864 Nov 14 20:42 time.db
-rw-r--r-- 1 dunegli fnalgrid        519 Nov 14 20:04 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid        519 Nov 14 18:50 g4_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid        519 Nov 14 20:42 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dunegli fnalgrid         52 Nov 14 18:40 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid          7 Nov 14 20:42 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid          0 Nov 14 20:07 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Nov 14 20:04 debugprod.log
justIN time: 2026-02-09 13:52:16 UTC       justIN version: 01.06.00