justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253907.0@dunegpschedd01.fnal.gov

Jobsub ID253907.0@dunegpschedd01.fnal.gov
Workflow ID10361
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-18 23:12:15
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-19 04:51:30
From worker nodeHostnamewn1905240.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-18 23:13:22
Input filesmonte-carlo-010361-000001
JobscriptExit code0
Real time5h (20197s)
CPU time5h (18467s = 91%)
Max RSS bytes4351860736 (4150 MiB)
Outputting started2025-11-19 04:50:01
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10361/1/001/reco_stage1.root
Finished2025-11-19 04:51:30
Saved logsjustin-logs:253907.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

igitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 172425 traces, 0 summary
Retagger: tagging trace set: gauss with 138233 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 138233 traces tagged "gauss"
FrameSaver: q=5.1503e+06 n=2514772 tag=gauss
wclsFrameSaver: saving 172425 traces tagged "wiener"
FrameSaver: q=5.40331e+06 n=2260883 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 54928 XUs and 44684 XVs -> 677 XUVs
C:0 T:1 142411 XUs and 113760 XVs -> 8907 XUVs
C:0 T:2 149306 XUs and 157445 XVs -> 6104 XUVs
C:0 T:3 71398 XUs and 66776 XVs -> 675 XUVs
C:0 T:4 16597 XUs and 16164 XVs -> 204 XUVs
C:0 T:5 55112 XUs and 42830 XVs -> 3945 XUVs
C:0 T:6 207463 XUs and 217879 XVs -> 8548 XUVs
C:0 T:7 77881 XUs and 79354 XVs -> 1153 XUVs
C:0 T:8 10720 XUs and 9747 XVs -> 244 XUVs
C:0 T:9 57132 XUs and 66879 XVs -> 6938 XUVs
C:0 T:10 22779 XUs and 35136 XVs -> 3804 XUVs
C:0 T:11 73 XUs and 35 XVs -> 6 XUVs
41205 XUVs total
14416 collection wire objects
36323 potential space points
Neighbour search...
1669833 tests to find 881512 neighbours
Iterating with no regularization...
Begin: 1.34296e+08
0 1.17566e+08
1 1.16385e+08
2 1.16253e+08
3 1.16218e+08
Now with regularization...
Begin: 9.98699e+07
0 9.99064e+07
Warning: metric increased
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410598569
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410598569
fLifetime: 30000 [us]
19-Nov-2025 04:49:58 GMT  Closed output file "reco_stage1.root"
19-Nov-2025 04:49:58 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          388.278       529.949       707.32        487.219       104.36         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00144562    0.00676308     0.0126851    0.00672299    0.00358879       10     
reco:ophit:OpHitFinder                              0.3136       0.399487      0.477531      0.406391      0.0516412       10     
reco:opflash:OpFlashFinder                          0.16152      0.228627      0.339761      0.218716      0.0502089       10     
reco:opslicer:OpSlicer                             0.651948       0.92063       1.26237      0.928656      0.184324        10     
reco:caldata:DataPrepModule                         5.99333       6.95382       8.44824       6.70738      0.772768        10     
reco:wclsdatasp:WireCellToolkit                     44.0175       55.1374       74.7129       49.5429       11.1151        10     
reco:gaushit:GausHitFinder                          1.17177       1.94137       2.72764       1.94319      0.395782        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00054783    0.00142135     0.0037052    0.000975003   0.000961557      10     
reco:reco3d:SpacePointSolver                        13.9753       26.1348       48.6156       22.2281       11.4924        10     
reco:hitpdune:DisambigFromSpacePoints               1.43289       2.75798       4.46806       2.55458      0.921667        10     
reco:pandora:StandardPandora                        35.322        57.9415       106.582       44.7628       25.0966        10     
reco:pandoraWriter:StandardPandora                 0.403856      0.715068       1.06715      0.695348       0.19757        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.22479       9.56784       21.0236       8.25338       4.26742        10     
reco:pandoraShower:LArPandoraShowerCreation         5.39648       10.8114       18.5414       10.6858       3.9936         10     
reco:pandoracalo:Calorimetry                        2.43024       4.7178        7.09093       4.74505       1.37906        10     
reco:pandoracalonosce:Calorimetry                   2.56623       4.26019       7.4943        4.04294       1.3432         10     
reco:pandorapid:Chi2ParticleID                     0.0068009     0.0084585     0.0110863    0.00826416    0.00113267       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0434813     0.0709531     0.125428      0.0678858     0.0230261       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00450376    0.00742632     0.0120194    0.00681558    0.00221477       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.88587       10.8169       16.5014       10.8067       3.32761        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.89846       10.1439       18.2351       8.98596       4.05446        10     
reco:emtrkmichelid:EmTrackMichelId                  179.535       246.049       304.984       239.907       37.8364        10     
reco:crttag:SingleCRTMatchingProducer              0.017374      0.0271042     0.047908      0.0238259    0.00938287       10     
reco:crtreco:TwoCRTMatchingProducer               0.00929369     0.0160973     0.0203701     0.0164662     0.0031843       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.819337       1.10271       1.31437       1.13947      0.150374        10     
reco:pandora2Track:LArPandoraTrackCreation          10.7524       23.3556       36.3865       22.3696       7.80115        10     
reco:pandora2calo:Calorimetry                       6.15532       12.1066       23.1328       11.0362       4.7392         10     
reco:pandora2calonosce:Calorimetry                  6.22732       11.8477       22.6215       10.2012       4.53836        10     
reco:pandora2pid:Chi2ParticleID                   0.00926746     0.0168031     0.0261866     0.0158757    0.00569267       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0483619     0.0835244     0.124771      0.0756473     0.0256294       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0113238     0.0183201     0.032095      0.0168061    0.00630085       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.28509       8.42197       16.0962       8.11295       3.30689        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.10851       10.3605       17.677        9.37241       3.25564        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.6274        8.80191       16.5663       8.59936       3.16804        10     
[art]:TriggerResults:TriggerResultInserter        7.2827e-05    0.000107244   0.000152787   9.0094e-05    2.9186e-05       10     
end_path:out1:RootOutput                           9.768e-06    1.33971e-05   2.4536e-05    1.1611e-05    4.34433e-06      10     
end_path:out1:RootOutput(write)                     2.66417       3.92278       4.82572       3.93345      0.665657        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5190.2 MB
  Peak resident set size usage (VmHWM): 4351.86 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.11 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10792588
-rw-r--r-- 1 dune01 dune01 2514350296 Nov 19 03:19 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2390016196 Nov 19 03:13 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2241087083 Nov 19 04:49 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1949683319 Nov 18 23:38 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1871348069 Nov 18 23:22 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   84491460 Nov 19 04:41 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242882 Nov 19 04:50 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 19 04:49 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 19 04:49 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 19 04:49 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 19 03:19 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 23:38 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 19 04:49 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 18 23:13 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 19 04:50 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 19 03:25 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 19 03:19 debugprod.log
justIN time: 2026-02-09 05:01:05 UTC       justIN version: 01.06.00