justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301516.1@dunegpschedd01.fnal.gov

Jobsub ID301516.1@dunegpschedd01.fnal.gov
Workflow ID12444
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-27 14:23:30
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-01-27 17:09:58
From worker nodeHostnamecomp19-05
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 14:24:14
Input filesmonte-carlo-012444-000001
JobscriptExit code0
Real time2h (9859s)
CPU time2h (9604s = 97%)
Max RSS bytes4122079232 (3931 MiB)
Outputting started2026-01-27 17:08:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12444/1/001/reco_stage1.root
Finished2026-01-27 17:09:58
Saved logsjustin-logs:301516.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 116731 traces, 0 summary
Retagger: tagging trace set: gauss with 89291 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 89291 traces tagged "gauss"
FrameSaver: q=2.93808e+06 n=1657177 tag=gauss
wclsFrameSaver: saving 116731 traces tagged "wiener"
FrameSaver: q=3.11958e+06 n=1477612 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 51738 XUs and 47188 XVs -> 1020 XUVs
C:0 T:1 95012 XUs and 86386 XVs -> 5226 XUVs
C:0 T:2 164785 XUs and 141424 XVs -> 9523 XUVs
C:0 T:4 9538 XUs and 7725 XVs -> 56 XUVs
C:0 T:5 39077 XUs and 30681 XVs -> 5118 XUVs
C:0 T:6 241264 XUs and 125706 XVs -> 43800 XUVs
C:0 T:7 43750 XUs and 24770 XVs -> 1161 XUVs
C:0 T:8 6073 XUs and 6182 XVs -> 37 XUVs
C:0 T:9 37638 XUs and 50479 XVs -> 4498 XUVs
C:0 T:10 152155 XUs and 53306 XVs -> 25786 XUVs
C:0 T:11 452 XUs and 534 XVs -> 103 XUVs
96328 XUVs total
12022 collection wire objects
89662 potential space points
Neighbour search...
27391036 tests to find 11034622 neighbours
Iterating with no regularization...
Begin: 4.36447e+07
0 3.42645e+07
1 3.36808e+07
2 3.35879e+07
3 3.35555e+07
Now with regularization...
Begin: 2.66559e+07
0 2.66013e+07
1 2.65465e+07
2 2.64989e+07
3 2.6456e+07
4 2.64167e+07
5 2.63818e+07
6 2.63511e+07
7 2.63244e+07
8 2.63011e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411999288
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411999288
fLifetime: 30000 [us]
27-Jan-2026 17:08:29 GMT  Closed output file "reco_stage1.root"
27-Jan-2026 17:08:29 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          212.065       308.98        471.11        292.035       73.2583        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000807299   0.00311562     0.0052505    0.00318215     0.0013146       10     
reco:ophit:OpHitFinder                             0.169868      0.217043      0.271398      0.215646      0.0265957       10     
reco:opflash:OpFlashFinder                         0.0850121     0.137743      0.174365       0.13962      0.0265548       10     
reco:opslicer:OpSlicer                             0.385193       0.49988      0.622918       0.49772      0.0729795       10     
reco:caldata:DataPrepModule                         5.0061        5.2943        5.93216       5.2125       0.269845        10     
reco:wclsdatasp:WireCellToolkit                     45.2733       49.6601       56.843        48.4538       3.4518         10     
reco:gaushit:GausHitFinder                         0.850378       1.15894       1.55123       1.12419      0.211067        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000459739   0.000607659   0.000978584   0.000589863   0.000140308      10     
reco:reco3d:SpacePointSolver                        5.3549        12.9917       30.6376       11.3793       7.85011        10     
reco:hitpdune:DisambigFromSpacePoints              0.943595       1.58707       2.61852       1.52255       0.49483        10     
reco:pandora:StandardPandora                        13.3812       29.4427       49.3346       26.2946       9.9531         10     
reco:pandoraWriter:StandardPandora                 0.318693      0.526533       0.8568       0.492914      0.175055        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.83271       4.15057       8.49248       4.03685       1.79092        10     
reco:pandoraShower:LArPandoraShowerCreation         1.57872       4.29112       10.0592       3.73742       2.29552        10     
reco:pandoracalo:Calorimetry                        1.03237       2.1015        4.43726       2.05456      0.929489        10     
reco:pandoracalonosce:Calorimetry                  0.872241       1.89441       4.13702       1.81405      0.890927        10     
reco:pandorapid:Chi2ParticleID                    0.00326265    0.00468147    0.00768931    0.00443805    0.00111021       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0389658     0.0518631     0.067108      0.0516854    0.00846317       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00330228    0.00467177    0.00801918    0.00433931    0.00119618       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.53515       4.31727       10.5736       3.81387       2.47734        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        1.465        4.15426       9.85844       3.67154       2.30491        10     
reco:emtrkmichelid:EmTrackMichelId                  106.693       144.916       200.533       139.62        27.876         10     
reco:crttag:SingleCRTMatchingProducer              0.0152346     0.019974      0.0310672      0.0193       0.0043069       10     
reco:crtreco:TwoCRTMatchingProducer                0.0075547     0.0109447     0.0155709     0.0103843    0.00260331       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.707839      0.921314       1.20494      0.895407       0.13735        10     
reco:pandora2Track:LArPandoraTrackCreation          5.22462       13.1449       26.3545       12.9094       5.71439        10     
reco:pandora2calo:Calorimetry                       2.38251       5.79187       11.7657       5.76066       2.57654        10     
reco:pandora2calonosce:Calorimetry                  2.19116       5.38012       10.9773       5.28885       2.42881        10     
reco:pandora2pid:Chi2ParticleID                   0.00687409     0.0113442     0.0170845     0.0111459    0.00254741       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0477488     0.349975       1.29431      0.0835037     0.429293        10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00692791     0.0112762     0.0168856     0.0110392    0.00249469       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.88738       4.21183       7.63409       4.27603       1.60384        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.96221       4.89032       9.27184       4.92581       2.06884        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.84295       4.4502        8.28811       4.44004       1.86135        10     
[art]:TriggerResults:TriggerResultInserter        4.7098e-05    6.71624e-05   0.000139852   6.03625e-05   2.53823e-05      10     
end_path:out1:RootOutput                          1.0649e-05    1.2789e-05    1.7611e-05    1.2412e-05    1.89271e-06      10     
end_path:out1:RootOutput(write)                     1.60919       2.26807       3.08022       2.20153      0.432962        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4935.42 MB
  Peak resident set size usage (VmHWM): 4122.08 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.19 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9069107
-rw-r--r-- 1 pltdune004 pltdune 2178634513 Jan 27 16:14 detsim_stage2.root
-rw-r--r-- 1 pltdune004 pltdune 2076070151 Jan 27 16:13 detsim_stage1.root
-rw-r--r-- 1 pltdune004 pltdune 1833706495 Jan 27 17:08 reco_stage1.root
-rw-r--r-- 1 pltdune004 pltdune 1589520204 Jan 27 14:39 g4_stage2.root
-rw-r--r-- 1 pltdune004 pltdune 1540399001 Jan 27 14:29 g4_stage1.root
-rw-r--r-- 1 pltdune004 pltdune   67870350 Jan 27 17:04 Pandora_Events.pndr
-rw-r--r-- 1 pltdune004 pltdune     242778 Jan 27 17:08 jobscript.log
-rw-r--r-- 1 pltdune004 pltdune     217088 Jan 27 17:08 mem.db
-rw-r--r-- 1 pltdune004 pltdune      62376 Jan 27 17:08 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune004 pltdune      36864 Jan 27 17:08 time.db
-rw-r--r-- 1 pltdune004 pltdune        519 Jan 27 16:15 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune004 pltdune        519 Jan 27 14:39 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune004 pltdune        519 Jan 27 17:08 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune004 pltdune         52 Jan 27 14:24 all-input-dids.txt
-rw-r--r-- 1 pltdune004 pltdune          7 Jan 27 17:08 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune004 pltdune          0 Jan 27 16:22 ProtoDUNE.log
-rw-r--r-- 1 pltdune004 pltdune          0 Jan 27 16:15 debugprod.log
justIN time: 2026-02-04 18:08:53 UTC       justIN version: 01.06.00