justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 256024.0@dunegpschedd01.fnal.gov

Jobsub ID256024.0@dunegpschedd01.fnal.gov
Workflow ID10557
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-24 04:11:37
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-24 08:08:27
From worker nodeHostnamewn-lb-08.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-24 04:14:34
Input filesmonte-carlo-010557-000001
JobscriptExit code0
Real time3h (13949s)
CPU time3h (13710s = 98%)
Max RSS bytes3915800576 (3734 MiB)
Outputting started2025-11-24 08:07:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10557/1/001/reco_stage1.root
Finished2025-11-24 08:08:27
Saved logsjustin-logs:256024.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

g tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 130037 traces, 0 summary
Retagger: tagging trace set: gauss with 97596 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 97596 traces tagged "gauss"
FrameSaver: q=3.36223e+06 n=1728948 tag=gauss
wclsFrameSaver: saving 130037 traces tagged "wiener"
FrameSaver: q=3.56839e+06 n=1528707 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 19192 XUs and 22466 XVs -> 182 XUVs
C:0 T:1 39994 XUs and 40364 XVs -> 3553 XUVs
C:0 T:2 44234 XUs and 39148 XVs -> 3578 XUVs
C:0 T:3 23038 XUs and 18217 XVs -> 31 XUVs
C:0 T:4 22493 XUs and 17891 XVs -> 206 XUVs
C:0 T:5 98522 XUs and 61163 XVs -> 8515 XUVs
C:0 T:6 81718 XUs and 92487 XVs -> 3641 XUVs
C:0 T:7 37372 XUs and 39659 XVs -> 810 XUVs
C:0 T:8 9983 XUs and 8453 XVs -> 81 XUVs
C:0 T:9 65344 XUs and 78969 XVs -> 4579 XUVs
C:0 T:10 48020 XUs and 23806 XVs -> 7329 XUVs
C:0 T:11 82 XUs and 51 XVs -> 17 XUVs
32522 XUVs total
11666 collection wire objects
30341 potential space points
Neighbour search...
1975575 tests to find 1061254 neighbours
Iterating with no regularization...
Begin: 5.63568e+07
0 4.73753e+07
1 4.66046e+07
2 4.64813e+07
3 4.64427e+07
Now with regularization...
Begin: 3.66079e+07
0 3.65759e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410698106
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410698106
fLifetime: 30000 [us]
24-Nov-2025 09:07:01 CET  Closed output file "reco_stage1.root"
24-Nov-2025 09:07:01 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          269.336       384.675        462.4        395.572       59.2952        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00109157    0.00501956    0.00739272    0.00554279     0.0021558       10     
reco:ophit:OpHitFinder                             0.293589      0.351651      0.380927      0.354912      0.0224385       10     
reco:opflash:OpFlashFinder                         0.131673       0.20312      0.236426      0.213298      0.0307924       10     
reco:opslicer:OpSlicer                              0.6176       0.716188      0.833895      0.719576      0.0590961       10     
reco:caldata:DataPrepModule                         6.1692        6.40568       7.0124        6.36988      0.220089        10     
reco:wclsdatasp:WireCellToolkit                     43.5574        53.2         60.1026       55.173        5.59598        10     
reco:gaushit:GausHitFinder                         0.851501       1.20476       1.4981        1.2005       0.192255        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000367721   0.000538606   0.000943113   0.000515382   0.000165042      10     
reco:reco3d:SpacePointSolver                        7.0799        14.6606       23.1319       13.9105       4.4017         10     
reco:hitpdune:DisambigFromSpacePoints              0.824362       1.40883       1.92192       1.42586      0.330752        10     
reco:pandora:StandardPandora                        15.2813       29.6288       40.9473       29.9504       7.50739        10     
reco:pandoraWriter:StandardPandora                 0.319495      0.434695      0.507015      0.442884      0.0606098       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.3488        4.5432        6.09637       4.82602       1.13796        10     
reco:pandoraShower:LArPandoraShowerCreation         2.43324       5.28661       8.19466       5.84226       1.66626        10     
reco:pandoracalo:Calorimetry                        1.3184        2.62795       3.47008       2.83729      0.688989        10     
reco:pandoracalonosce:Calorimetry                   1.2083        2.45992       3.46968       2.65128      0.673464        10     
reco:pandorapid:Chi2ParticleID                    0.00426459    0.00550148    0.00710406     0.0054418    0.000717548      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0373863     0.0512063     0.0583592     0.0533331    0.00656049       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00429771    0.00525675    0.00609899    0.00536825    0.000514376      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.39281       5.58595       9.05764       6.06017       1.95177        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.27237       5.2833        8.70612       5.59541       1.86026        10     
reco:emtrkmichelid:EmTrackMichelId                  145.85        198.055       240.743       203.316       29.1603        10     
reco:crttag:SingleCRTMatchingProducer             0.00991899     0.0159641     0.0261859     0.0142266    0.00496858       10     
reco:crtreco:TwoCRTMatchingProducer               0.00765402     0.0116495     0.0149154     0.0125748    0.00245209       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.730834       1.0022        1.16616       1.03017      0.126386        10     
reco:pandora2Track:LArPandoraTrackCreation          7.1763        15.2118       21.4856       15.748        4.47544        10     
reco:pandora2calo:Calorimetry                       3.42485       7.68349       10.4662       8.12543       2.36052        10     
reco:pandora2calonosce:Calorimetry                  3.22487       7.26629       9.51814       8.02946       2.1857         10     
reco:pandora2pid:Chi2ParticleID                    0.0100058     0.0125682     0.0143831     0.0129872    0.00140346       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0505003     0.0662827     0.0751556     0.0692789    0.00802225       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00989414     0.0125987     0.0143077     0.0129627    0.00142617       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.72409       5.60549       7.33068       6.22329       1.49803        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.98047       6.5032        8.35769       7.24364       1.8675         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.7761        5.84516       7.47484       6.14597       1.68917        10     
[art]:TriggerResults:TriggerResultInserter         3.992e-05    5.47978e-05   0.000132108   4.57835e-05   2.62618e-05      10     
end_path:out1:RootOutput                           7.412e-06    9.1408e-06    1.5673e-05    8.5425e-06    2.22615e-06      10     
end_path:out1:RootOutput(write)                     2.34745       3.10938       3.79124       3.20518      0.411318        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4716.46 MB
  Peak resident set size usage (VmHWM): 3915.8 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.99 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9005532
-rw-r--r--. 1 dune009 dune 2158538681 Nov 24 08:01 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2053727608 Nov 24 07:59 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 1819249341 Nov 24 09:07 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1593657612 Nov 24 05:35 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1527195838 Nov 24 05:23 g4_stage1.root
-rw-r--r--. 1 dune009 dune   68685010 Nov 24 09:01 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242867 Nov 24 09:07 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 24 09:07 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 24 09:07 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 24 09:07 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 24 08:01 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 24 05:35 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 24 09:07 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 24 05:16 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 24 09:07 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 24 08:07 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 24 08:01 debugprod.log
justIN time: 2026-02-08 20:09:22 UTC       justIN version: 01.06.00