justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249057.1@dunegpschedd02.fnal.gov

Jobsub ID249057.1@dunegpschedd02.fnal.gov
Workflow ID10442
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-21 00:17:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-21 03:28:24
From worker nodeHostnamewn-da-25.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-21 00:18:29
Input filesmonte-carlo-010442-000001
JobscriptExit code0
Real time3h (11315s)
CPU time3h (11183s = 98%)
Max RSS bytes3902668800 (3721 MiB)
Outputting started2025-11-21 03:27:06
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10442/1/001/reco_stage1.root
Finished2025-11-21 03:28:24
Saved logsjustin-logs:249057.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 114068 traces, 0 summary
Retagger: tagging trace set: gauss with 84754 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84754 traces tagged "gauss"
FrameSaver: q=2.78199e+06 n=1488220 tag=gauss
wclsFrameSaver: saving 114068 traces tagged "wiener"
FrameSaver: q=2.95376e+06 n=1296267 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 30879 XUs and 19364 XVs -> 854 XUVs
C:0 T:1 170198 XUs and 84093 XVs -> 26678 XUVs
C:0 T:2 54269 XUs and 82528 XVs -> 5393 XUVs
C:0 T:3 26641 XUs and 28867 XVs -> 126 XUVs
C:0 T:4 12985 XUs and 9662 XVs -> 143 XUVs
C:0 T:5 58901 XUs and 36013 XVs -> 3687 XUVs
C:0 T:6 60018 XUs and 104697 XVs -> 4165 XUVs
C:0 T:7 22601 XUs and 32098 XVs -> 208 XUVs
C:0 T:8 8972 XUs and 7277 XVs -> 254 XUVs
C:0 T:9 98600 XUs and 191535 XVs -> 32825 XUVs
C:0 T:10 26850 XUs and 23384 XVs -> 3229 XUVs
C:0 T:11 200 XUs and 189 XVs -> 18 XUVs
77580 XUVs total
11670 collection wire objects
71983 potential space points
Neighbour search...
21347907 tests to find 8541764 neighbours
Iterating with no regularization...
Begin: 4.89872e+07
0 3.81376e+07
1 3.74975e+07
2 3.74034e+07
3 3.73702e+07
Now with regularization...
Begin: 3.06204e+07
0 3.05558e+07
1 3.0506e+07
2 3.04639e+07
3 3.0427e+07
4 3.03957e+07
5 3.03692e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410639649
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410639649
fLifetime: 30000 [us]
21-Nov-2025 04:27:04 CET  Closed output file "reco_stage1.root"
21-Nov-2025 04:27:04 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          323.565       366.041       437.47        345.318       41.3404        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000801665   0.00415078    0.00788538    0.00427695    0.00216435       10     
reco:ophit:OpHitFinder                             0.213118      0.247716      0.286078      0.245889      0.0218556       10     
reco:opflash:OpFlashFinder                         0.112567      0.147444      0.181595      0.150201      0.0221801       10     
reco:opslicer:OpSlicer                             0.444322       0.59736      0.671828      0.603577      0.0665178       10     
reco:caldata:DataPrepModule                         4.62991       4.73458       5.08938       4.6882       0.133269        10     
reco:wclsdatasp:WireCellToolkit                     36.9719       52.1345       67.4916       51.1885       10.3669        10     
reco:gaushit:GausHitFinder                          0.99661       1.19036       1.4307        1.1784       0.171517        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000426451   0.000675003   0.00082469    0.000697161   0.000117254      10     
reco:reco3d:SpacePointSolver                        9.65484       15.5059       30.5575       13.8755       5.86374        10     
reco:hitpdune:DisambigFromSpacePoints              0.961352       1.35379       1.93125       1.23479      0.323174        10     
reco:pandora:StandardPandora                        19.5117       32.5786       50.7641       30.4181       8.70542        10     
reco:pandoraWriter:StandardPandora                 0.329893      0.387575      0.467874      0.377815      0.0430104       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.16324       3.87252       4.74804       3.78696      0.479406        10     
reco:pandoraShower:LArPandoraShowerCreation         2.73865       4.52512       7.26581       4.34997       1.33904        10     
reco:pandoracalo:Calorimetry                        1.43174       2.07849       2.48201       2.11807      0.288683        10     
reco:pandoracalonosce:Calorimetry                   1.30044        1.906        2.28611       1.94089      0.268907        10     
reco:pandorapid:Chi2ParticleID                    0.00471949    0.00532056    0.00715636    0.00517143    0.00064239       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0351267     0.0461125     0.0571409     0.0443671    0.00639701       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00463893    0.00506902     0.005594     0.00509047    0.000291619      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.20919       4.59981       7.13104       4.70809       1.37108        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.22521       4.38052       6.4489        4.3647        1.27389        10     
reco:emtrkmichelid:EmTrackMichelId                  159.284       192.509       231.596       189.002       22.4564        10     
reco:crttag:SingleCRTMatchingProducer              0.0068189     0.0143323     0.0260888     0.0123707    0.00535811       10     
reco:crtreco:TwoCRTMatchingProducer               0.00558794    0.00926458     0.0136849    0.00956767    0.00226604       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.551068      0.779588       1.01732      0.772196       0.1442         10     
reco:pandora2Track:LArPandoraTrackCreation          7.87666       12.9037       20.7387       11.9333       3.86159        10     
reco:pandora2calo:Calorimetry                       3.35799       6.28704       9.44013       5.97007       1.72624        10     
reco:pandora2calonosce:Calorimetry                  3.08907       5.76926       8.48681       5.5624        1.55175        10     
reco:pandora2pid:Chi2ParticleID                    0.0118326     0.0132917     0.0168365     0.0126815    0.00152525       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0470329     0.059434      0.0747198     0.0580232    0.00850308       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0115058     0.0131837     0.0165954     0.0125352    0.00155508       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.18178       4.65288       6.86038       4.38897       1.19715        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.88036       5.21676       6.96132       5.20858       1.26531        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.56773       4.60828       6.02647       4.67226       1.03869        10     
[art]:TriggerResults:TriggerResultInserter        2.9785e-05    3.90341e-05   9.9949e-05    3.2151e-05    2.03616e-05      10     
end_path:out1:RootOutput                           5.851e-06    7.6459e-06    1.4788e-05    6.4975e-06    2.69246e-06      10     
end_path:out1:RootOutput(write)                     2.23056       2.76546       3.43082       2.66892      0.364918        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4706.17 MB
  Peak resident set size usage (VmHWM): 3902.67 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.93 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9380948
-rw-r--r--. 1 dune009 dune 2230185656 Nov 21 03:24 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2118447692 Nov 21 03:22 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 1912596930 Nov 21 04:27 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1670725016 Nov 21 01:34 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1599640259 Nov 21 01:24 g4_stage1.root
-rw-r--r--. 1 dune009 dune   73881850 Nov 21 04:23 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243108 Nov 21 04:27 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 21 04:27 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 21 04:27 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 21 04:27 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 21 03:24 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 21 01:34 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 21 04:27 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 21 01:18 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 21 04:27 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 21 03:31 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 21 03:24 debugprod.log
justIN time: 2026-02-09 01:56:10 UTC       justIN version: 01.06.00