justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233011.0@dunegpschedd02.fnal.gov

Jobsub ID233011.0@dunegpschedd02.fnal.gov
Workflow ID9348
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-26 07:14:19
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-10-26 11:52:26
From worker nodeHostnamewn-lb-08.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-26 07:15:26
Input filesmonte-carlo-009348-000001
JobscriptExit code0
Real time4h (16519s)
CPU time4h (16348s = 98%)
Max RSS bytes4551528448 (4340 MiB)
Outputting started2025-10-26 11:50:46
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09348/1/001/reco_stage1.root
Finished2025-10-26 11:52:26
Saved logsjustin-logs:233011.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ver: saving 111852 traces tagged "gauss"
FrameSaver: q=5.22502e+06 n=2049986 tag=gauss
wclsFrameSaver: saving 143092 traces tagged "wiener"
FrameSaver: q=5.60754e+06 n=1850310 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 66341 XUs and 69159 XVs -> 2991 XUVs
C:0 T:1 380229 XUs and 481870 XVs -> 69296 XUVs
C:0 T:2 144863 XUs and 126562 XVs -> 7131 XUVs
C:0 T:3 60538 XUs and 46585 XVs -> 802 XUVs
C:0 T:4 13768 XUs and 18670 XVs -> 1114 XUVs
C:0 T:5 44478 XUs and 51780 XVs -> 4792 XUVs
C:0 T:6 78848 XUs and 56565 XVs -> 4140 XUVs
C:0 T:7 32737 XUs and 22123 XVs -> 223 XUVs
C:0 T:8 12322 XUs and 11915 XVs -> 130 XUVs
C:0 T:9 68246 XUs and 86304 XVs -> 4703 XUVs
C:0 T:10 17505 XUs and 17860 XVs -> 3846 XUVs
C:0 T:11 298 XUs and 259 XVs -> 40 XUVs
99208 XUVs total
14717 collection wire objects
83498 potential space points
Neighbour search...
37939156 tests to find 8373440 neighbours
Iterating with no regularization...
Begin: 6.51587e+08
0 1.904e+08
1 1.65902e+08
2 1.60069e+08
3 1.57365e+08
4 1.55837e+08
5 1.54867e+08
6 1.54217e+08
7 1.53775e+08
8 1.53445e+08
9 1.53193e+08
10 1.52996e+08
11 1.52831e+08
12 1.52693e+08
Now with regularization...
Begin: 9.86089e+07
0 9.61964e+07
1 9.45561e+07
2 9.32102e+07
3 9.20562e+07
4 9.11124e+07
5 9.03144e+07
6 8.96356e+07
7 8.90608e+07
8 8.85773e+07
9 8.81523e+07
10 8.777e+07
11 8.74448e+07
12 8.71629e+07
13 8.69128e+07
14 8.66964e+07
15 8.65082e+07
16 8.63447e+07
17 8.62084e+07
18 8.60891e+07
19 8.59845e+07
20 8.58905e+07
21 8.58081e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410122346
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410122346
fLifetime: 30000 [us]
26-Oct-2025 12:50:44 CET  Closed output file "reco_stage1.root"
26-Oct-2025 12:50:44 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          333.227       495.735       636.059       486.432       96.183         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00124537    0.00657028     0.0110959    0.00675593    0.00339246       10     
reco:ophit:OpHitFinder                              0.34819      0.428715      0.495404      0.433931      0.0471307       10     
reco:opflash:OpFlashFinder                         0.254218      0.321966      0.405817      0.329039      0.0439112       10     
reco:opslicer:OpSlicer                             0.628896      0.859762       1.05829      0.887033      0.126436        10     
reco:caldata:DataPrepModule                         4.93106       6.27331       7.0082        6.46185      0.532148        10     
reco:wclsdatasp:WireCellToolkit                     44.1923       56.1655       66.6782       58.779        8.00698        10     
reco:gaushit:GausHitFinder                         0.906825       1.45889       1.79574       1.47779      0.236876        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000536486   0.000891278   0.00120328    0.000930235   0.000239945      10     
reco:reco3d:SpacePointSolver                        15.3046       24.6209       53.3466       19.1382       11.9326        10     
reco:hitpdune:DisambigFromSpacePoints               1.11605       2.11037       2.90825       2.10563      0.523947        10     
reco:pandora:StandardPandora                        21.2997       57.2539        115.2        52.9998       26.0583        10     
reco:pandoraWriter:StandardPandora                 0.397503      0.595973      0.949583      0.586167      0.142417        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.40437       10.2271       15.5499       10.0626       3.38869        10     
reco:pandoraShower:LArPandoraShowerCreation         5.24301       11.2654       19.1202       10.2782       4.6978         10     
reco:pandoracalo:Calorimetry                        2.94872       5.42867       8.08153       5.31595       1.73204        10     
reco:pandoracalonosce:Calorimetry                   3.02852       4.98156       7.84984       4.72081       1.58797        10     
reco:pandorapid:Chi2ParticleID                    0.00530897    0.00712026     0.0101794    0.00666261    0.00144243       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0426495     0.0565943     0.0687319     0.0581348     0.0101376       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0051629    0.00663288    0.00810555    0.00660217    0.000986268      10     
reco:pandoraShowercalo:ShowerCalorimetry            6.48857       11.2077       17.8758       10.221        4.07184        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.89725       11.044        17.0466       11.2238       3.85465        10     
reco:emtrkmichelid:EmTrackMichelId                  144.112       211.17        257.241       209.743       35.4221        10     
reco:crttag:SingleCRTMatchingProducer              0.0143656     0.0195003     0.0264864     0.0194024    0.00385191       10     
reco:crtreco:TwoCRTMatchingProducer                0.0102417     0.0152369     0.0193731     0.0153789    0.00309358       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.720708       1.00408       1.29836       0.92659      0.196192        10     
reco:pandora2Track:LArPandoraTrackCreation          14.1678       25.3698       41.6284       24.2687       7.34514        10     
reco:pandora2calo:Calorimetry                       7.70957       12.9505       22.9599       11.4654       3.95494        10     
reco:pandora2calonosce:Calorimetry                  7.06392       11.5851       17.8545       10.7756       2.81082        10     
reco:pandora2pid:Chi2ParticleID                   0.00997842     0.0150782     0.0199798     0.0146215     0.0029766       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0498843     0.0724515     0.090762      0.0734402     0.0128234       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00983031     0.0149946     0.0200239     0.0146527    0.00310496       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.07895       8.13454       11.8247       7.92136       1.67668        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.14531       9.38707       14.0325       9.26505       1.98172        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.95444       7.87477       11.1903       7.81404       1.54945        10     
[art]:TriggerResults:TriggerResultInserter        3.7968e-05    5.57154e-05   0.000137937   4.2233e-05    2.90966e-05      10     
end_path:out1:RootOutput                           7.641e-06    8.9882e-06    1.4361e-05     8.247e-06    1.86939e-06      10     
end_path:out1:RootOutput(write)                     2.40442       3.53394       4.42993       3.77639      0.624657        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5354.78 MB
  Peak resident set size usage (VmHWM): 4551.53 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.91 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11503624
-rw-r--r--. 1 dune009 dune 2672887615 Oct 26 11:25 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2538528915 Oct 26 11:23 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2385321180 Oct 26 12:50 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2088695625 Oct 26 08:37 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2011951907 Oct 26 08:23 g4_stage1.root
-rw-r--r--. 1 dune009 dune   81710770 Oct 26 12:43 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243332 Oct 26 12:50 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 26 12:50 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 26 12:50 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 26 12:50 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 26 11:25 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 26 08:37 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 26 12:50 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 26 08:15 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 26 12:50 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 26 11:33 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 26 11:25 debugprod.log
justIN time: 2025-11-05 23:36:28 UTC       justIN version: 01.05.01