justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263485.1@dunegpschedd01.fnal.gov

Jobsub ID263485.1@dunegpschedd01.fnal.gov
Workflow ID10999
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-04 16:45:43
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-12-04 20:18:19
From worker nodeHostnamewn-da-10.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-04 16:46:36
Input filesmonte-carlo-010999-000001
JobscriptExit code0
Real time3h (12621s)
CPU time3h (12514s = 99%)
Max RSS bytes4359991296 (4158 MiB)
Outputting started2025-12-04 20:16:57
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10999/1/001/reco_stage1.root
Finished2025-12-04 20:18:19
Saved logsjustin-logs:263485.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

l adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 108434 traces, 0 summary
Retagger: tagging trace set: gauss with 88370 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88370 traces tagged "gauss"
FrameSaver: q=3.06165e+06 n=1708167 tag=gauss
wclsFrameSaver: saving 108434 traces tagged "wiener"
FrameSaver: q=3.21712e+06 n=1556077 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 63530 XUs and 48122 XVs -> 1100 XUVs
C:0 T:1 153531 XUs and 150292 XVs -> 12250 XUVs
C:0 T:2 112516 XUs and 150016 XVs -> 4173 XUVs
C:0 T:3 64653 XUs and 70800 XVs -> 791 XUVs
C:0 T:4 15623 XUs and 11450 XVs -> 233 XUVs
C:0 T:5 64627 XUs and 37880 XVs -> 5274 XUVs
C:0 T:6 42210 XUs and 49152 XVs -> 1655 XUVs
C:0 T:7 18660 XUs and 22820 XVs -> 107 XUVs
C:0 T:8 3246 XUs and 3888 XVs -> 29 XUVs
C:0 T:9 22930 XUs and 58104 XVs -> 5734 XUVs
C:0 T:10 14248 XUs and 6847 XVs -> 1367 XUVs
C:0 T:11 112 XUs and 58 XVs -> 3 XUVs
32716 XUVs total
9793 collection wire objects
29886 potential space points
Neighbour search...
3007696 tests to find 1465864 neighbours
Iterating with no regularization...
Begin: 5.19632e+07
0 4.40822e+07
1 4.3522e+07
2 4.34294e+07
3 4.33994e+07
Now with regularization...
Begin: 3.64735e+07
0 3.64542e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410913703
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410913703
fLifetime: 30000 [us]
04-Dec-2025 21:16:55 CET  Closed output file "reco_stage1.root"
04-Dec-2025 21:16:55 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          302.89        394.68        531.826       389.686       63.0715        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00117866    0.00490751    0.00855213    0.00505833     0.0026311       10     
reco:ophit:OpHitFinder                             0.246078      0.285542      0.322807       0.28155      0.0191984       10     
reco:opflash:OpFlashFinder                         0.137892      0.190644      0.227031       0.19362      0.0286196       10     
reco:opslicer:OpSlicer                              0.55892      0.699339      0.876455      0.711623      0.0848012       10     
reco:caldata:DataPrepModule                         4.82672       5.00784       5.20588       4.99294      0.100481        10     
reco:wclsdatasp:WireCellToolkit                     35.8455       56.7708       72.6624       56.936        12.2324        10     
reco:gaushit:GausHitFinder                         0.996954       1.26498       1.55706       1.21462      0.163706        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000428706   0.000692646   0.000986729   0.000708003   0.000172415      10     
reco:reco3d:SpacePointSolver                        8.47104       13.4592       22.5144       12.6872       4.5741         10     
reco:hitpdune:DisambigFromSpacePoints               1.03006       1.52169       2.23611       1.4613       0.339149        10     
reco:pandora:StandardPandora                        20.0011       34.2032       49.4842       34.9899       9.45518        10     
reco:pandoraWriter:StandardPandora                 0.333384      0.422774      0.546437      0.421274      0.0593759       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.07572       4.90952       7.44208       4.64548       1.45375        10     
reco:pandoraShower:LArPandoraShowerCreation         2.11303       5.36166       10.3266       4.29393       2.36342        10     
reco:pandoracalo:Calorimetry                        1.52881       2.62712       3.85074       2.4597       0.775771        10     
reco:pandoracalonosce:Calorimetry                   1.39903       2.45288       3.71505       2.28638      0.732757        10     
reco:pandorapid:Chi2ParticleID                    0.00516489    0.00625732    0.00851572    0.00620084    0.000897854      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0433568     0.0497554     0.0665814     0.0468903    0.00726206       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00512754    0.00607379    0.00696294    0.00615853    0.000554437      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.88856       5.19732       9.42519       4.35689       2.14648        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.85244       4.99762       9.30079       4.20046       2.09076        10     
reco:emtrkmichelid:EmTrackMichelId                  167.588       207.526       266.506       203.656       26.5487        10     
reco:crttag:SingleCRTMatchingProducer             0.00969048     0.0132176     0.0177975     0.0129798    0.00223285       10     
reco:crtreco:TwoCRTMatchingProducer               0.00642166     0.0101994     0.0165729     0.0096998    0.00262695       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.615248      0.843247       1.16462      0.810308      0.175798        10     
reco:pandora2Track:LArPandoraTrackCreation          8.24727       14.4401       26.3118       13.711        4.90592        10     
reco:pandora2calo:Calorimetry                       3.70609       6.88586       12.4993       6.47274       2.41281        10     
reco:pandora2calonosce:Calorimetry                  3.47428       6.50653       12.0867       6.1115        2.32089        10     
reco:pandora2pid:Chi2ParticleID                    0.0121385     0.0148869     0.0175471     0.0146238    0.00186701       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0535817     0.063651      0.0845144     0.0629576    0.00903498       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0119861     0.0146935     0.0170355     0.0145796     0.0017688       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.25874       5.24486       9.87219       4.99269       1.75818        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.16188       5.64075       10.3046       5.35997       1.89071        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.83933       4.98261       9.14495       4.7174        1.65642        10     
[art]:TriggerResults:TriggerResultInserter        2.8945e-05    4.39415e-05   0.000120719   3.7421e-05    2.58777e-05      10     
end_path:out1:RootOutput                           6.553e-06    7.2979e-06     1.016e-05     6.988e-06    9.93336e-07      10     
end_path:out1:RootOutput(write)                     2.36344       2.89816       3.78299       2.78056      0.406786        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5162.7 MB
  Peak resident set size usage (VmHWM): 4359.99 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.11 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10274152
-rw-r--r--. 1 dune009 dune 2413323247 Dec  4 20:09 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2291660040 Dec  4 20:06 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2096753484 Dec  4 21:16 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1857522620 Dec  4 18:05 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1786105449 Dec  4 17:53 g4_stage1.root
-rw-r--r--. 1 dune009 dune   74754590 Dec  4 21:12 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242901 Dec  4 21:16 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Dec  4 21:16 mem.db
-rw-r--r--. 1 dune009 dune      62376 Dec  4 21:16 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Dec  4 21:16 time.db
-rw-r--r--. 1 dune009 dune        519 Dec  4 20:09 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec  4 18:05 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec  4 21:16 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Dec  4 17:46 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Dec  4 21:16 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Dec  4 20:15 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Dec  4 20:09 debugprod.log
justIN time: 2025-12-18 18:07:18 UTC       justIN version: 01.05.03