justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269489.1@dunegpschedd01.fnal.gov

Jobsub ID269489.1@dunegpschedd01.fnal.gov
Workflow ID11298
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-21 09:33:10
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-12-21 14:37:20
From worker nodeHostnamewn-la-19.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-21 09:46:02
Input filesmonte-carlo-011298-000001
JobscriptExit code0
Real time4h (17382s)
CPU time4h (17221s = 99%)
Max RSS bytes4860559360 (4635 MiB)
Outputting started2025-12-21 14:35:44
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11298/1/001/reco_stage1.root
Finished2025-12-21 14:37:20
Saved logsjustin-logs:269489.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

igitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 126099 traces, 0 summary
Retagger: tagging trace set: gauss with 96947 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96947 traces tagged "gauss"
FrameSaver: q=3.30217e+06 n=1745436 tag=gauss
wclsFrameSaver: saving 126099 traces tagged "wiener"
FrameSaver: q=3.49178e+06 n=1554156 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33756 XUs and 39003 XVs -> 658 XUVs
C:0 T:1 66881 XUs and 71788 XVs -> 7440 XUVs
C:0 T:2 78976 XUs and 77426 XVs -> 7399 XUVs
C:0 T:3 29121 XUs and 29002 XVs -> 271 XUVs
C:0 T:4 10558 XUs and 13852 XVs -> 158 XUVs
C:0 T:5 40487 XUs and 39290 XVs -> 4305 XUVs
C:0 T:6 145426 XUs and 118546 XVs -> 13338 XUVs
C:0 T:7 40907 XUs and 38498 XVs -> 667 XUVs
C:0 T:8 10119 XUs and 9569 XVs -> 151 XUVs
C:0 T:9 65334 XUs and 71332 XVs -> 5018 XUVs
C:0 T:10 97550 XUs and 41640 XVs -> 18701 XUVs
C:0 T:11 67 XUs and 40 XVs -> 13 XUVs
58119 XUVs total
11968 collection wire objects
54283 potential space points
Neighbour search...
11556963 tests to find 4912214 neighbours
Iterating with no regularization...
Begin: 5.51747e+07
0 4.6702e+07
1 4.60334e+07
2 4.59268e+07
3 4.5892e+07
Now with regularization...
Begin: 3.69193e+07
0 3.68639e+07
1 3.68228e+07
2 3.67892e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411250897
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411250897
fLifetime: 30000 [us]
21-Dec-2025 15:35:42 CET  Closed output file "reco_stage1.root"
21-Dec-2025 15:35:42 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          418.59        497.273       609.765       488.956       53.6204        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00144558    0.00637629     0.0100997    0.00615418    0.00267246       10     
reco:ophit:OpHitFinder                             0.399613      0.439356      0.476433      0.444458      0.0273627       10     
reco:opflash:OpFlashFinder                         0.263907      0.326884      0.375195      0.334907      0.0337251       10     
reco:opslicer:OpSlicer                             0.765171      0.914038       1.12248      0.896108      0.108645        10     
reco:caldata:DataPrepModule                         6.36809       6.52392       7.10517       6.45663      0.201674        10     
reco:wclsdatasp:WireCellToolkit                     43.9559       55.587        65.9031       57.1689       8.06533        10     
reco:gaushit:GausHitFinder                          1.32609       1.52435       1.85164       1.49816      0.160683        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000448439   0.000755749   0.00101585    0.000772361   0.000195275      10     
reco:reco3d:SpacePointSolver                        14.9373       21.2584       32.6561       20.1469       4.44461        10     
reco:hitpdune:DisambigFromSpacePoints               1.49543       2.03504       2.55125       2.0825       0.347026        10     
reco:pandora:StandardPandora                        33.3646       45.6366       64.5293       42.9222       9.30057        10     
reco:pandoraWriter:StandardPandora                 0.465217       0.5563       0.618171      0.574696      0.0480056       10     
reco:pandoraTrack:LArPandoraTrackCreation           6.41618       8.45297       12.1883       7.61711       1.79416        10     
reco:pandoraShower:LArPandoraShowerCreation         5.88219       9.25136       12.4617       8.96122       2.09105        10     
reco:pandoracalo:Calorimetry                        3.63765       4.57518       6.47418       4.23347      0.845062        10     
reco:pandoracalonosce:Calorimetry                   3.38119       4.45162       6.16574       4.26244       0.81731        10     
reco:pandorapid:Chi2ParticleID                    0.00624446    0.00701206    0.00832193    0.00696595    0.000609318      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0537947     0.0623107     0.0680987     0.0642665    0.00467868       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00609187    0.00678494    0.00728027    0.00693679    0.000403135      10     
reco:pandoraShowercalo:ShowerCalorimetry            5.94706       9.93311       15.0552       9.39397       2.63765        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.53975       10.0301       14.1474       9.20861       2.62199        10     
reco:emtrkmichelid:EmTrackMichelId                  205.175       229.376       252.749       235.622       16.1246        10     
reco:crttag:SingleCRTMatchingProducer              0.0162296     0.025453      0.0351877     0.0280427    0.00671744       10     
reco:crtreco:TwoCRTMatchingProducer                0.0141511     0.020501      0.028163      0.0194522    0.00455433       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.993338       1.2106        1.47876       1.2253       0.127655        10     
reco:pandora2Track:LArPandoraTrackCreation          18.9761       25.5714       35.5653       25.122        4.69396        10     
reco:pandora2calo:Calorimetry                       9.77543       13.6923       21.3306       12.6107       3.25412        10     
reco:pandora2calonosce:Calorimetry                  8.9157        12.4188       16.9839       12.3031       2.23759        10     
reco:pandora2pid:Chi2ParticleID                    0.0120242     0.0154374     0.0190361     0.0157181    0.00177305       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0675529     0.0781776     0.0877605     0.0772758    0.00616394       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0125562     0.015395      0.0190683     0.0153523     0.001636        10     
reco:pandora2Shower:LArPandoraShowerCreation        7.50949       9.37133       11.6879       9.53797       1.39833        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.4415        10.572        14.1351       10.0496       1.89204        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.72245       9.2919        11.673        9.24948       1.70532        10     
[art]:TriggerResults:TriggerResultInserter        4.1201e-05    6.74282e-05   0.000164514   5.7195e-05    3.33271e-05      10     
end_path:out1:RootOutput                           8.242e-06    9.5491e-06    1.5623e-05    8.9035e-06    2.08623e-06      10     
end_path:out1:RootOutput(write)                     3.27706       3.77706       4.31247       3.8726       0.308598        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5663.32 MB
  Peak resident set size usage (VmHWM): 4860.56 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.08 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11172416
-rw-r--r--. 1 dune009 dune 2588831784 Dec 21 14:10 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2457739565 Dec 21 14:07 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2305748669 Dec 21 15:35 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2045240603 Dec 21 11:10 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1960477887 Dec 21 10:54 g4_stage1.root
-rw-r--r--. 1 dune009 dune   81894360 Dec 21 15:30 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243007 Dec 21 15:35 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Dec 21 15:35 mem.db
-rw-r--r--. 1 dune009 dune      62376 Dec 21 15:35 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Dec 21 15:35 time.db
-rw-r--r--. 1 dune009 dune        519 Dec 21 14:10 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 21 11:10 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 21 15:35 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Dec 21 10:46 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Dec 21 15:35 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Dec 21 14:18 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Dec 21 14:10 debugprod.log
justIN time: 2026-02-07 06:34:14 UTC       justIN version: 01.06.00