justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237989.0@dunegpschedd02.fnal.gov

Jobsub ID237989.0@dunegpschedd02.fnal.gov
Workflow ID9645
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-05 20:01:47
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-06 01:15:13
From worker nodeHostnamewn-la-17.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 20:02:38
Input filesmonte-carlo-009645-000001
JobscriptExit code0
Real time5h (18644s)
CPU time5h (18495s = 99%)
Max RSS bytes4601245696 (4388 MiB)
Outputting started2025-11-06 01:13:23
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09645/1/001/reco_stage1.root
Finished2025-11-06 01:15:13
Saved logsjustin-logs:237989.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 135299 traces, 0 summary
Retagger: tagging trace set: gauss with 103771 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 103771 traces tagged "gauss"
FrameSaver: q=3.5687e+06 n=1815215 tag=gauss
wclsFrameSaver: saving 135299 traces tagged "wiener"
FrameSaver: q=3.76562e+06 n=1602718 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 30665 XUs and 30377 XVs -> 362 XUVs
C:0 T:1 69568 XUs and 60585 XVs -> 6061 XUVs
C:0 T:2 91634 XUs and 139066 XVs -> 6109 XUVs
C:0 T:3 45121 XUs and 64570 XVs -> 428 XUVs
C:0 T:4 17767 XUs and 15229 XVs -> 162 XUVs
C:0 T:5 111290 XUs and 58632 XVs -> 10530 XUVs
C:0 T:6 97053 XUs and 98116 XVs -> 7212 XUVs
C:0 T:7 42145 XUs and 41345 XVs -> 476 XUVs
C:0 T:8 5630 XUs and 4096 XVs -> 113 XUVs
C:0 T:9 32861 XUs and 29859 XVs -> 2518 XUVs
C:0 T:10 27958 XUs and 22551 XVs -> 5829 XUVs
C:0 T:11 634 XUs and 527 XVs -> 92 XUVs
39892 XUVs total
11730 collection wire objects
37084 potential space points
Neighbour search...
4375406 tests to find 2099128 neighbours
Iterating with no regularization...
Begin: 6.75137e+07
0 5.78812e+07
1 5.71954e+07
2 5.71043e+07
3 5.70776e+07
Now with regularization...
Begin: 4.67207e+07
0 4.66873e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410334221
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410334221
fLifetime: 30000 [us]
06-Nov-2025 02:13:21 CET  Closed output file "reco_stage1.root"
06-Nov-2025 02:13:21 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          409.794       611.763       803.193       619.338       118.353        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00132862    0.00704011     0.0111613    0.00740293     0.0030847       10     
reco:ophit:OpHitFinder                              0.27898      0.360995      0.414098      0.364239      0.0408307       10     
reco:opflash:OpFlashFinder                         0.171568       0.2879       0.384833      0.289176      0.0572321       10     
reco:opslicer:OpSlicer                             0.561192      0.751851      0.887074      0.751348      0.104473        10     
reco:caldata:DataPrepModule                         4.96435       5.14701       5.5704        5.09684      0.181707        10     
reco:wclsdatasp:WireCellToolkit                     41.6576       51.4633       56.2406       53.4427       4.81374        10     
reco:gaushit:GausHitFinder                          0.91369       1.33655       1.73659       1.37045      0.218088        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00108077    0.00160644    0.00195504    0.00164853    0.000294934      10     
reco:reco3d:SpacePointSolver                        10.3367       23.2894       37.8172       21.9479       7.95083        10     
reco:hitpdune:DisambigFromSpacePoints               1.21731       2.17999       3.28322       2.13018      0.611728        10     
reco:pandora:StandardPandora                        26.3553       50.1262       76.2789       53.6966       16.026         10     
reco:pandoraWriter:StandardPandora                 0.572972      0.752758       0.94695      0.739665      0.115404        10     
reco:pandoraTrack:LArPandoraTrackCreation           10.5512       19.2432       28.2641       18.0223       5.30541        10     
reco:pandoraShower:LArPandoraShowerCreation         12.395        24.8738       44.3373       23.1474       8.93134        10     
reco:pandoracalo:Calorimetry                        4.46257       7.88294       11.4956       7.61513       1.92526        10     
reco:pandoracalonosce:Calorimetry                   4.26294       7.53756       10.1124       7.65301       1.72216        10     
reco:pandorapid:Chi2ParticleID                    0.00486818    0.00684605     0.0104118    0.00661423    0.00149918       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0455183     0.0598711     0.0837723     0.0574199     0.0112309       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00531123     0.0068645     0.0101757    0.00619577    0.00148078       10     
reco:pandoraShowercalo:ShowerCalorimetry            9.49877       18.8484       30.5847       17.734        6.21824        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       9.0375        18.7387       30.1977       17.5453       6.42312        10     
reco:emtrkmichelid:EmTrackMichelId                  147.623       201.12         250.9        202.051       31.7581        10     
reco:crttag:SingleCRTMatchingProducer              0.0248562     0.0400874     0.0523412     0.0407179    0.00877959       10     
reco:crtreco:TwoCRTMatchingProducer                0.0259974     0.040445      0.0527672     0.0405774    0.00749408       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.807031       1.04863       1.39099       1.05057      0.171603        10     
reco:pandora2Track:LArPandoraTrackCreation          38.8484       63.7612       92.5814       62.3455       16.6083        10     
reco:pandora2calo:Calorimetry                       15.4902       24.7035       35.239        24.2138       6.12482        10     
reco:pandora2calonosce:Calorimetry                  16.031        23.787        33.1293       24.2763       5.36224        10     
reco:pandora2pid:Chi2ParticleID                    0.0107117     0.0147701     0.0204784     0.0141782    0.00258239       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0580171     0.0763588     0.0983511     0.0736274     0.0129407       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0112348     0.0147707     0.0197339     0.0142149    0.00263512       10     
reco:pandora2Shower:LArPandoraShowerCreation        14.4287       22.317        30.1654        22.17        4.53616        10     
reco:pandora2Showercalo:ShowerCalorimetry           14.3882       20.0242       26.2312       20.7691       3.28846        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      12.1046       18.0719       22.6377       19.1376       3.4397         10     
[art]:TriggerResults:TriggerResultInserter         4.028e-05    6.72908e-05   0.00014743    6.1758e-05    2.85734e-05      10     
end_path:out1:RootOutput                           7.461e-06    9.7824e-06    1.6354e-05     9.188e-06    2.58512e-06      10     
end_path:out1:RootOutput(write)                     2.55151       3.43978       4.40538       3.41818      0.591418        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5403.88 MB
  Peak resident set size usage (VmHWM): 4601.25 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.13 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11615112
-rw-r--r--. 1 dune009 dune 2667106321 Nov  6 00:29 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2524931902 Nov  6 00:27 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2433675021 Nov  6 02:13 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2130743790 Nov  5 21:31 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2043362458 Nov  5 21:11 g4_stage1.root
-rw-r--r--. 1 dune009 dune   93441830 Nov  6 02:07 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242690 Nov  6 02:13 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  6 02:13 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  6 02:13 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  6 02:13 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  6 00:29 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  5 21:31 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  6 02:13 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  5 21:02 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  6 02:13 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  6 00:38 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  6 00:29 debugprod.log
justIN time: 2026-02-11 06:06:51 UTC       justIN version: 01.06.00