justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 288563.0@dunegpschedd02.fnal.gov

Jobsub ID288563.0@dunegpschedd02.fnal.gov
Workflow ID12118
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-20 14:48:56
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-01-20 17:53:30
From worker nodeHostnameb9p09p5455.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-20 14:53:04
Input filesmonte-carlo-012118-000001
JobscriptExit code0
Real time2h (10724s)
CPU time2h (9931s = 92%)
Max RSS bytes4365393920 (4163 MiB)
Outputting started2026-01-20 17:51:49
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12118/1/001/reco_stage1.root
Finished2026-01-20 17:53:30
Saved logsjustin-logs:288563.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

unning tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 163197 traces, 0 summary
Retagger: tagging trace set: gauss with 124551 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 124551 traces tagged "gauss"
FrameSaver: q=4.31565e+06 n=2189483 tag=gauss
wclsFrameSaver: saving 163197 traces tagged "wiener"
FrameSaver: q=4.55379e+06 n=1932445 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 18704 XUs and 19328 XVs -> 235 XUVs
C:0 T:1 23166 XUs and 22210 XVs -> 2103 XUVs
C:0 T:2 167935 XUs and 197543 XVs -> 14206 XUVs
C:0 T:3 69341 XUs and 69840 XVs -> 857 XUVs
C:0 T:4 13358 XUs and 15013 XVs -> 94 XUVs
C:0 T:5 48120 XUs and 40243 XVs -> 4023 XUVs
C:0 T:6 141469 XUs and 129918 XVs -> 8057 XUVs
C:0 T:7 54737 XUs and 42596 XVs -> 507 XUVs
C:0 T:8 12932 XUs and 11360 XVs -> 134 XUVs
C:0 T:9 86603 XUs and 111623 XVs -> 7740 XUVs
C:0 T:10 57810 XUs and 32948 XVs -> 3909 XUVs
C:0 T:11 257 XUs and 227 XVs -> 22 XUVs
41887 XUVs total
14580 collection wire objects
37788 potential space points
Neighbour search...
3035470 tests to find 1532568 neighbours
Iterating with no regularization...
Begin: 7.63907e+07
0 6.43802e+07
1 6.35407e+07
2 6.34178e+07
3 6.338e+07
Now with regularization...
Begin: 5.11616e+07
0 5.11361e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411853228
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411853228
fLifetime: 30000 [us]
20-Jan-2026 18:51:48 CET  Closed output file "reco_stage1.root"
20-Jan-2026 18:51:48 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          207.136       261.386       405.688       247.846       54.566         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000872579   0.00317321     0.0052495    0.00318119    0.00138685       10     
reco:ophit:OpHitFinder                             0.216602      0.258886      0.289715      0.264433      0.0250261       10     
reco:opflash:OpFlashFinder                         0.133481      0.187246      0.239622      0.195198      0.0334021       10     
reco:opslicer:OpSlicer                             0.463572       0.54551      0.717616      0.526626      0.0768501       10     
reco:caldata:DataPrepModule                         3.72636       3.90818       4.16158       3.87864      0.134749        10     
reco:wclsdatasp:WireCellToolkit                     24.6435       25.8226       27.3719       25.7896      0.768636        10     
reco:gaushit:GausHitFinder                         0.748107      0.901591       1.19713       0.87229      0.143257        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000318817   0.000413705   0.000625613   0.000407144   8.72315e-05      10     
reco:reco3d:SpacePointSolver                        7.54571       13.203        25.7971       13.1433       4.78459        10     
reco:hitpdune:DisambigFromSpacePoints              0.853069       1.38405       2.33073       1.28507      0.417773        10     
reco:pandora:StandardPandora                        17.2015       29.7419       85.0812       24.5043       18.9586        10     
reco:pandoraWriter:StandardPandora                 0.260267      0.324241      0.440361      0.315208      0.0518147       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.95856       2.94481       4.54959       2.94928      0.714764        10     
reco:pandoraShower:LArPandoraShowerCreation          1.869        3.13291       6.16127       2.87192       1.15889        10     
reco:pandoracalo:Calorimetry                        1.21453       1.76738       3.0749        1.68442      0.503658        10     
reco:pandoracalonosce:Calorimetry                   1.16935       1.61396       2.46707       1.56088      0.365018        10     
reco:pandorapid:Chi2ParticleID                    0.00336151    0.00434811    0.00596627     0.0042997    0.000672488      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0310189     0.0376864     0.0470267     0.0384181    0.00461476       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00316357    0.00404212    0.00474331    0.00397742    0.000499524      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.16605       3.45631       6.26164       3.25923       1.13471        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.28058       3.48283       6.26711       3.16457       1.18729        10     
reco:emtrkmichelid:EmTrackMichelId                  113.523       135.22        173.81        130.981       17.5525        10     
reco:crttag:SingleCRTMatchingProducer             0.00743223     0.0103853     0.0145519      0.01086     0.00212427       10     
reco:crtreco:TwoCRTMatchingProducer               0.00475194    0.00613955    0.00835446    0.00593188    0.000937009      10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.525475       0.76337      0.936503      0.777047      0.116142        10     
reco:pandora2Track:LArPandoraTrackCreation          6.40653       9.20975       16.7532       8.29611       2.95496        10     
reco:pandora2calo:Calorimetry                       3.40809       4.96032       8.15871       4.61357       1.3539         10     
reco:pandora2calonosce:Calorimetry                  3.09994       4.66978       7.89664       4.24382       1.31988        10     
reco:pandora2pid:Chi2ParticleID                   0.00776768     0.0103865     0.0133551    0.00976253    0.00171236       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0393702     0.0491603     0.0635847     0.0476943    0.00648703       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00778306    0.00990456     0.0130304    0.00951483     0.0015265       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.27718       3.28232       5.72445       2.94406      0.941356        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.99714       4.24255       6.79352       3.97668       1.06557        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.65864       3.88819       6.16385       3.61019      0.972205        10     
[art]:TriggerResults:TriggerResultInserter         3.752e-05    5.78751e-05   0.000168783   4.24355e-05   3.80306e-05      10     
end_path:out1:RootOutput                           6.44e-06     9.8013e-06    1.9401e-05     9.24e-06     3.40297e-06      10     
end_path:out1:RootOutput(write)                     1.84478       2.25553       2.93835       2.16525      0.352763        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5210.31 MB
  Peak resident set size usage (VmHWM): 4365.39 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.47 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11035252
-rw-r--r--. 1 duneprd np-comp 2557227827 Jan 20 18:05 detsim_stage2.root
-rw-r--r--. 1 duneprd np-comp 2423823229 Jan 20 18:03 detsim_stage1.root
-rw-r--r--. 1 duneprd np-comp 2267925964 Jan 20 18:51 reco_stage1.root
-rw-r--r--. 1 duneprd np-comp 2025767489 Jan 20 16:13 g4_stage2.root
-rw-r--r--. 1 duneprd np-comp 1943552473 Jan 20 15:59 g4_stage1.root
-rw-r--r--. 1 duneprd np-comp   81182880 Jan 20 18:48 Pandora_Events.pndr
-rw-r--r--. 1 duneprd np-comp     243139 Jan 20 18:51 jobscript.log
-rw-r--r--. 1 duneprd np-comp     217088 Jan 20 18:51 mem.db
-rw-r--r--. 1 duneprd np-comp      62376 Jan 20 18:51 Pandora_Geometry.xml
-rw-r--r--. 1 duneprd np-comp      36864 Jan 20 18:51 time.db
-rw-r--r--. 1 duneprd np-comp        519 Jan 20 18:05 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 duneprd np-comp        519 Jan 20 16:13 g4_protoDUNE_hist.root
-rw-r--r--. 1 duneprd np-comp        519 Jan 20 18:51 protoDUNE_reco1_hist.root
-rw-r--r--. 1 duneprd np-comp         52 Jan 20 15:53 all-input-dids.txt
-rw-r--r--. 1 duneprd np-comp          7 Jan 20 18:51 justin-processed-pfns.txt
-rw-r--r--. 1 duneprd np-comp          0 Jan 20 18:10 ProtoDUNE.log
-rw-r--r--. 1 duneprd np-comp          0 Jan 20 18:05 debugprod.log
justIN time: 2026-02-04 07:13:33 UTC       justIN version: 01.06.00