justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266395.0@dunegpschedd01.fnal.gov

Jobsub ID266395.0@dunegpschedd01.fnal.gov
Workflow ID11144
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-10 23:21:29
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-12-11 03:12:54
From worker nodeHostnamewn1905250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 23:23:08
Input filesmonte-carlo-011144-000001
JobscriptExit code0
Real time3h (13686s)
CPU time3h (13313s = 97%)
Max RSS bytes4344836096 (4143 MiB)
Outputting started2025-12-11 03:11:16
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11144/1/001/reco_stage1.root
Finished2025-12-11 03:12:54
Saved logsjustin-logs:266395.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 136705 traces, 0 summary
Retagger: tagging trace set: gauss with 102278 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 102278 traces tagged "gauss"
FrameSaver: q=3.91199e+06 n=1766586 tag=gauss
wclsFrameSaver: saving 136705 traces tagged "wiener"
FrameSaver: q=4.17771e+06 n=1567570 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 46140 XUs and 60381 XVs -> 1001 XUVs
C:0 T:1 126179 XUs and 124725 XVs -> 17878 XUVs
C:0 T:2 92174 XUs and 88263 XVs -> 7411 XUVs
C:0 T:3 44189 XUs and 41227 XVs -> 782 XUVs
C:0 T:4 16171 XUs and 14321 XVs -> 168 XUVs
C:0 T:5 63147 XUs and 42468 XVs -> 7751 XUVs
C:0 T:6 25206 XUs and 23450 XVs -> 908 XUVs
C:0 T:7 12368 XUs and 11451 XVs -> 91 XUVs
C:0 T:8 9575 XUs and 8193 XVs -> 85 XUVs
C:0 T:9 73767 XUs and 68189 XVs -> 9177 XUVs
C:0 T:10 30429 XUs and 21886 XVs -> 2873 XUVs
C:0 T:11 91 XUs and 52 XVs -> 9 XUVs
48134 XUVs total
10132 collection wire objects
44568 potential space points
Neighbour search...
7901280 tests to find 3330758 neighbours
Iterating with no regularization...
Begin: 1.07767e+08
0 9.36217e+07
1 9.23138e+07
2 9.21254e+07
3 9.20745e+07
Now with regularization...
Begin: 7.83848e+07
0 7.83333e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411041198
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411041198
fLifetime: 30000 [us]
11-Dec-2025 03:11:13 GMT  Closed output file "reco_stage1.root"
11-Dec-2025 03:11:13 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          338.504       430.329       522.263       425.984       54.2675        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000968383    0.005437      0.0100691    0.00555413    0.00284572       10     
reco:ophit:OpHitFinder                             0.318441      0.354832      0.421512       0.34024      0.0351418       10     
reco:opflash:OpFlashFinder                         0.209616      0.250802      0.282036      0.247321      0.0222376       10     
reco:opslicer:OpSlicer                              0.62521      0.800529       0.99457      0.791058      0.127697        10     
reco:caldata:DataPrepModule                         4.74407       6.06099       7.51442       5.93568      0.929336        10     
reco:wclsdatasp:WireCellToolkit                     36.7232       42.1388       47.1794       41.3623       3.53856        10     
reco:gaushit:GausHitFinder                          1.03534       1.38771       1.84813       1.34695      0.247005        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000381788   0.000642023   0.000763277   0.000683386   0.000112193      10     
reco:reco3d:SpacePointSolver                        12.9108       20.5427       27.4984       20.904        5.46627        10     
reco:hitpdune:DisambigFromSpacePoints               1.58716       2.17238       3.20704       2.15742      0.462927        10     
reco:pandora:StandardPandora                        25.6327       42.968        60.5704       43.4618        10.66         10     
reco:pandoraWriter:StandardPandora                 0.413948      0.499301      0.686596      0.473083      0.0832971       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.59035       4.65741       6.28225       4.34627      0.839809        10     
reco:pandoraShower:LArPandoraShowerCreation         3.31155       5.6203         7.795        5.22068       1.39443        10     
reco:pandoracalo:Calorimetry                        1.93458       2.95356       4.58495       2.88729       0.78495        10     
reco:pandoracalonosce:Calorimetry                   2.05902       2.84083       4.14386       2.84787      0.557349        10     
reco:pandorapid:Chi2ParticleID                    0.00500564     0.0070129    0.00879878    0.00743407    0.00132082       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0473848     0.072191      0.0882406     0.0742242     0.0140359       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00430892    0.00678813    0.00910897    0.00709701    0.00150548       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.51824       6.23301       8.87364       5.82772       1.41502        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.27419       6.17284       7.98991       6.09945       1.25273        10     
reco:emtrkmichelid:EmTrackMichelId                   172.3        225.768       272.614       226.397       32.5408        10     
reco:crttag:SingleCRTMatchingProducer              0.0165505     0.0207431     0.0318009     0.0175157     0.0050166       10     
reco:crtreco:TwoCRTMatchingProducer               0.00792682     0.0113201     0.0176281     0.010343     0.00265851       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.04132       1.16284       1.4056        1.14146      0.103388        10     
reco:pandora2Track:LArPandoraTrackCreation          10.8503       17.555        24.9702       16.8271       3.94385        10     
reco:pandora2calo:Calorimetry                       5.38891       8.89478       12.2058       8.40497       1.97868        10     
reco:pandora2calonosce:Calorimetry                  4.68456       8.19346       11.9547       8.56141       1.92393        10     
reco:pandora2pid:Chi2ParticleID                   0.00993127     0.0161908     0.0237565     0.0152013    0.00456322       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0554192     0.0796731     0.133072      0.0682875     0.0254458       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0102568     0.0158973     0.0267163     0.0133649    0.00526575       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.45504       5.89371       8.12716       6.12499       1.44683        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.01219       7.03344       8.44317       7.89993       1.43516        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.80345       6.33985       7.65617       6.76229       1.13201        10     
[art]:TriggerResults:TriggerResultInserter        4.3382e-05    6.62288e-05   0.000116329   6.28985e-05   2.02676e-05      10     
end_path:out1:RootOutput                           9.678e-06    1.21479e-05    1.587e-05    1.18225e-05   1.76842e-06      10     
end_path:out1:RootOutput(write)                     2.54471       3.47251       3.92585       3.57472       0.41871        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5183.46 MB
  Peak resident set size usage (VmHWM): 4344.84 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.76 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11325012
-rw-r--r-- 1 dune01 dune01 2620535101 Dec 11 01:57 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2479366996 Dec 11 01:54 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2348554953 Dec 11 03:11 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2079014557 Dec 10 23:48 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1984212917 Dec 10 23:29 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   84497510 Dec 11 03:06 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242880 Dec 11 03:11 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec 11 03:11 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec 11 03:11 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec 11 03:11 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec 11 01:57 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 10 23:48 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 11 03:11 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec 10 23:23 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec 11 03:11 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec 11 02:03 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec 11 01:57 debugprod.log
justIN time: 2026-02-07 13:30:28 UTC       justIN version: 01.06.00