justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 247058.1@dunegpschedd02.fnal.gov

Jobsub ID247058.1@dunegpschedd02.fnal.gov
Workflow ID10254
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-16 10:29:47
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-16 13:53:39
From worker nodeHostnamewn-la-07.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-16 10:30:56
Input filesmonte-carlo-010254-000001
JobscriptExit code0
Real time3h (12021s)
CPU time3h (11888s = 98%)
Max RSS bytes4088438784 (3899 MiB)
Outputting started2025-11-16 13:51:17
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10254/1/001/reco_stage1.root
Finished2025-11-16 13:53:39
Saved logsjustin-logs:247058.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

cKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 151216 traces, 0 summary
Retagger: tagging trace set: gauss with 115498 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 115498 traces tagged "gauss"
FrameSaver: q=4.19159e+06 n=2065031 tag=gauss
wclsFrameSaver: saving 151216 traces tagged "wiener"
FrameSaver: q=4.43304e+06 n=1831134 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33888 XUs and 40693 XVs -> 393 XUVs
C:0 T:1 89269 XUs and 103127 XVs -> 6196 XUVs
C:0 T:2 26456 XUs and 42280 XVs -> 1188 XUVs
C:0 T:3 14860 XUs and 20428 XVs -> 121 XUVs
C:0 T:4 13077 XUs and 11210 XVs -> 213 XUVs
C:0 T:5 53081 XUs and 47338 XVs -> 7449 XUVs
C:0 T:6 92238 XUs and 88036 XVs -> 3684 XUVs
C:0 T:7 36772 XUs and 34909 XVs -> 305 XUVs
C:0 T:8 15714 XUs and 15629 XVs -> 230 XUVs
C:0 T:9 109491 XUs and 135279 XVs -> 7194 XUVs
C:0 T:10 84391 XUs and 50729 XVs -> 9859 XUVs
C:0 T:11 386 XUs and 242 XVs -> 33 XUVs
36865 XUVs total
13000 collection wire objects
34092 potential space points
Neighbour search...
2434926 tests to find 1200468 neighbours
Iterating with no regularization...
Begin: 8.44961e+07
0 6.61217e+07
1 6.47897e+07
2 6.46155e+07
3 6.45699e+07
Now with regularization...
Begin: 5.21972e+07
0 5.21653e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410547359
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410547359
fLifetime: 30000 [us]
16-Nov-2025 14:51:16 CET  Closed output file "reco_stage1.root"
16-Nov-2025 14:51:16 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          268.59        327.508       392.177       328.108       45.1702        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000919619   0.00457448    0.00772539    0.00526882    0.00203077       10     
reco:ophit:OpHitFinder                             0.268811      0.309505      0.394349      0.299208      0.0389018       10     
reco:opflash:OpFlashFinder                         0.162773       0.1977       0.258493      0.185517      0.0312857       10     
reco:opslicer:OpSlicer                             0.541226      0.629942      0.820803       0.60387      0.0840971       10     
reco:caldata:DataPrepModule                         4.70865       5.16908       6.41163       4.8217       0.603401        10     
reco:wclsdatasp:WireCellToolkit                     34.0647       47.4735       63.5158       48.2771       8.58097        10     
reco:gaushit:GausHitFinder                          0.82225       1.08207       1.37503       1.08491      0.160718        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000302354   0.000435012   0.000695773   0.000409865   0.000111871      10     
reco:reco3d:SpacePointSolver                        9.83771       13.7334       21.2108       12.8875       3.09434        10     
reco:hitpdune:DisambigFromSpacePoints              0.854574       1.28836       1.93613       1.32696      0.315452        10     
reco:pandora:StandardPandora                        18.3332       27.4169       43.5405       25.8693       8.44637        10     
reco:pandoraWriter:StandardPandora                 0.310745      0.391367      0.461292      0.410261      0.0467483       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.23506       4.40369       7.01276       4.00972       1.08924        10     
reco:pandoraShower:LArPandoraShowerCreation         2.86754       4.80254       7.05745       4.65074       1.25746        10     
reco:pandoracalo:Calorimetry                        1.4143         2.329        3.76746       2.15832       0.64597        10     
reco:pandoracalonosce:Calorimetry                   1.26801       2.23686       3.89082       2.01571      0.702975        10     
reco:pandorapid:Chi2ParticleID                    0.00356643    0.00472267    0.00563343    0.00469053    0.00063408       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0307031     0.0414502     0.0480087     0.0424691    0.00525692       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00356642    0.00449984    0.00559952     0.0043754    0.000566839      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.21927       4.88757       7.44382       4.70441       1.45415        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.50792       4.6062        6.90182       4.39178       1.42324        10     
reco:emtrkmichelid:EmTrackMichelId                  136.198       159.958       187.92        162.566       15.8088        10     
reco:crttag:SingleCRTMatchingProducer             0.00807019     0.0118166     0.016191      0.0116475    0.00248496       10     
reco:crtreco:TwoCRTMatchingProducer               0.00611012    0.00968836     0.0111177     0.0103855     0.0015615       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.609701        0.784        1.00415      0.815055      0.124751        10     
reco:pandora2Track:LArPandoraTrackCreation          7.45651       13.9107       18.7948       12.7812       3.62436        10     
reco:pandora2calo:Calorimetry                       3.18913       6.81697       9.7387        6.45629       2.06056        10     
reco:pandora2calonosce:Calorimetry                  3.08867        6.243        9.03293       5.81279       1.74353        10     
reco:pandora2pid:Chi2ParticleID                   0.00862973     0.0106357     0.0125881     0.0108844    0.00132005       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0409303     0.052202      0.0628397     0.0538997     0.0066463       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00869301     0.0107271     0.0128645     0.0110286    0.00137655       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.99142       5.04809       7.07103       4.86002       1.13794        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.17734       5.79078       8.07087       5.27027       1.61708        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.07802       5.05719       6.43888       4.88171       1.11454        10     
[art]:TriggerResults:TriggerResultInserter        3.1237e-05    4.52196e-05   0.000106058   3.7732e-05    2.12233e-05      10     
end_path:out1:RootOutput                           5.918e-06    7.2667e-06    1.2389e-05     6.64e-06     1.80014e-06      10     
end_path:out1:RootOutput(write)                     2.17014       2.62375       2.96025       2.65012      0.239556        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4893.99 MB
  Peak resident set size usage (VmHWM): 4088.44 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.23 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.20 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9830940
-rw-r--r--. 1 dune009 dune 2324633795 Nov 16 13:55 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2208498794 Nov 16 13:53 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2003169926 Nov 16 14:51 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1769503625 Nov 16 11:49 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1687264173 Nov 16 11:36 g4_stage1.root
-rw-r--r--. 1 dune009 dune   73197650 Nov 16 14:46 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242807 Nov 16 14:51 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 16 14:51 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 16 14:51 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 16 14:51 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 16 13:55 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 16 11:49 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 16 14:51 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 16 11:30 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 16 14:51 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 16 13:59 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 16 13:55 debugprod.log
justIN time: 2026-02-09 10:52:27 UTC       justIN version: 01.06.00