justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271303.0@dunegpschedd01.fnal.gov

Jobsub ID271303.0@dunegpschedd01.fnal.gov
Workflow ID11467
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-02 09:24:17
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-01-02 13:24:15
From worker nodeHostnamecn519.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-02 09:49:02
Input filesmonte-carlo-011467-000001
JobscriptExit code0
Real time3h (12707s)
CPU time3h (12547s = 98%)
Max RSS bytes4294832128 (4095 MiB)
Outputting started2026-01-02 13:20:50
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11467/1/001/reco_stage1.root
Finished2026-01-02 13:24:15
Saved logsjustin-logs:271303.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ol adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 119828 traces, 0 summary
Retagger: tagging trace set: gauss with 90579 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90579 traces tagged "gauss"
FrameSaver: q=3.08643e+06 n=1572437 tag=gauss
wclsFrameSaver: saving 119828 traces tagged "wiener"
FrameSaver: q=3.26416e+06 n=1387178 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 34202 XUs and 36117 XVs -> 555 XUVs
C:0 T:1 64431 XUs and 61939 XVs -> 5001 XUVs
C:0 T:2 88963 XUs and 124219 XVs -> 4501 XUVs
C:0 T:3 48503 XUs and 49420 XVs -> 301 XUVs
C:0 T:4 5353 XUs and 9795 XVs -> 50 XUVs
C:0 T:5 26328 XUs and 52731 XVs -> 6756 XUVs
C:0 T:6 81752 XUs and 102312 XVs -> 4964 XUVs
C:0 T:7 30151 XUs and 30901 XVs -> 251 XUVs
C:0 T:8 5594 XUs and 8346 XVs -> 107 XUVs
C:0 T:9 46404 XUs and 93574 XVs -> 9410 XUVs
C:0 T:10 34723 XUs and 24189 XVs -> 2951 XUVs
C:0 T:11 52 XUs and 30 XVs -> 7 XUVs
34854 XUVs total
10666 collection wire objects
32574 potential space points
Neighbour search...
3209244 tests to find 1632810 neighbours
Iterating with no regularization...
Begin: 6.11472e+07
0 5.00461e+07
1 4.93495e+07
2 4.92434e+07
3 4.92083e+07
Now with regularization...
Begin: 4.08746e+07
0 4.08295e+07
1 4.08042e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411492023
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411492023
fLifetime: 30000 [us]
02-Jan-2026 13:20:47 GMT  Closed output file "reco_stage1.root"
02-Jan-2026 13:20:47 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          376.284       493.673       835.518       434.974       141.359        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00106803    0.00322549    0.00635348    0.00316866    0.00135527       10     
reco:ophit:OpHitFinder                             0.294261      0.337707      0.405934      0.324342      0.0346671       10     
reco:opflash:OpFlashFinder                         0.206558      0.258428       0.33572      0.240983      0.0395096       10     
reco:opslicer:OpSlicer                             0.657028      0.808527      0.985453      0.809552      0.116947        10     
reco:caldata:DataPrepModule                         6.62972       7.15307       8.52989       6.97734      0.544944        10     
reco:wclsdatasp:WireCellToolkit                     54.5962       57.4384       61.4274       56.5965       2.23219        10     
reco:gaushit:GausHitFinder                          1.34183       1.66726       2.37216       1.53983      0.354714        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000613098   0.000780307   0.00100587    0.000740781   0.000127913      10     
reco:reco3d:SpacePointSolver                        9.49752       17.3253       45.8892       13.5413       11.0635        10     
reco:hitpdune:DisambigFromSpacePoints               1.67146       2.43914       4.53626       2.00473       0.9371         10     
reco:pandora:StandardPandora                        27.4674       46.0088       126.068       32.7854       28.9697        10     
reco:pandoraWriter:StandardPandora                 0.449978      0.564911      0.805101      0.528253      0.111255        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.85291       9.34806       15.436        8.30621       2.69935        10     
reco:pandoraShower:LArPandoraShowerCreation         5.7081        10.1304       22.0067       8.40653       4.85067        10     
reco:pandoracalo:Calorimetry                        3.1982        4.33221       6.77106       4.00715       1.22139        10     
reco:pandoracalonosce:Calorimetry                   2.84578       3.9306        6.33142       3.42073       1.16768        10     
reco:pandorapid:Chi2ParticleID                    0.00559632    0.00679416    0.00817888    0.00655103    0.000864394      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0642718     0.0759528     0.108028      0.0699076     0.014894        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00539324    0.00655875    0.00805012    0.00650472    0.000744981      10     
reco:pandoraShowercalo:ShowerCalorimetry            5.22887       9.17685       20.8258       7.73675       4.64922        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.88917       9.0903        20.406        7.68637       4.70921        10     
reco:emtrkmichelid:EmTrackMichelId                  181.118       222.969       307.819       206.46        43.0801        10     
reco:crttag:SingleCRTMatchingProducer              0.0192791     0.0283784     0.0408336     0.0279007    0.00646174       10     
reco:crtreco:TwoCRTMatchingProducer                0.0155055     0.0233371     0.0353135     0.0229828    0.00612154       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.21449       1.56587       2.39932       1.43268       0.34982        10     
reco:pandora2Track:LArPandoraTrackCreation          19.1984       30.9599       61.7248       26.0826       12.907         10     
reco:pandora2calo:Calorimetry                       7.27665       12.8228       26.2591       11.2186       5.85303        10     
reco:pandora2calonosce:Calorimetry                  6.91029       12.2064       26.0266       10.6032       5.7066         10     
reco:pandora2pid:Chi2ParticleID                    0.0135126     0.0156992     0.0209749     0.0148205    0.00254928       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.074303      0.0950713     0.141617      0.0866734     0.0200724       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0135934     0.0155683     0.0203746     0.0149687    0.00234295       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.26543       9.56189       22.0935       8.17421       4.86955        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.3795        10.6512       23.0051       9.43805       5.05593        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.77119       9.20158       21.4076       7.84081       4.71787        10     
[art]:TriggerResults:TriggerResultInserter        5.0612e-05    6.99504e-05   0.000130948   6.4927e-05    2.15482e-05      10     
end_path:out1:RootOutput                           9.862e-06    1.26199e-05   1.9477e-05    1.17595e-05   2.64809e-06      10     
end_path:out1:RootOutput(write)                     2.72822       3.29301       4.98018       3.01844      0.689086        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5098.36 MB
  Peak resident set size usage (VmHWM): 4294.83 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.26 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.50 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.25 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9989820
-rw-r--r-- 1 pildune32 pildune 2338647973 Jan  2 11:56 detsim_stage2.root
-rw-r--r-- 1 pildune32 pildune 2218244707 Jan  2 11:53 detsim_stage1.root
-rw-r--r-- 1 pildune32 pildune 2018633584 Jan  2 13:20 reco_stage1.root
-rw-r--r-- 1 pildune32 pildune 1816918470 Jan  2 10:13 g4_stage2.root
-rw-r--r-- 1 pildune32 pildune 1763605861 Jan  2 09:56 g4_stage1.root
-rw-r--r-- 1 pildune32 pildune   72913960 Jan  2 13:15 Pandora_Events.pndr
-rw-r--r-- 1 pildune32 pildune     243087 Jan  2 13:20 jobscript.log
-rw-r--r-- 1 pildune32 pildune     217088 Jan  2 13:20 mem.db
-rw-r--r-- 1 pildune32 pildune      62376 Jan  2 13:20 Pandora_Geometry.xml
-rw-r--r-- 1 pildune32 pildune      36864 Jan  2 13:20 time.db
-rw-r--r-- 1 pildune32 pildune        519 Jan  2 11:56 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Jan  2 10:13 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Jan  2 13:20 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune32 pildune         52 Jan  2 09:49 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune          7 Jan  2 13:20 justin-processed-pfns.txt
-rw-r--r-- 1 pildune32 pildune          0 Jan  2 12:03 ProtoDUNE.log
-rw-r--r-- 1 pildune32 pildune          0 Jan  2 11:56 debugprod.log
justIN time: 2026-02-06 22:22:28 UTC       justIN version: 01.06.00