justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265385.1@dunegpschedd01.fnal.gov

Jobsub ID265385.1@dunegpschedd01.fnal.gov
Workflow ID11122
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-09 13:45:51
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-12-09 21:36:50
From worker nodeHostnamewn2210250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-09 17:15:12
Input filesmonte-carlo-011122-000001
JobscriptExit code0
Real time4h (15602s)
CPU time4h (15100s = 96%)
Max RSS bytes4359278592 (4157 MiB)
Outputting started2025-12-09 21:35:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11122/1/001/reco_stage1.root
Finished2025-12-09 21:36:50
Saved logsjustin-logs:265385.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Finder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 145465 traces, 0 summary
Retagger: tagging trace set: gauss with 105931 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 105931 traces tagged "gauss"
FrameSaver: q=3.75056e+06 n=1812569 tag=gauss
wclsFrameSaver: saving 145465 traces tagged "wiener"
FrameSaver: q=4.00137e+06 n=1579432 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 30467 XUs and 15693 XVs -> 218 XUVs
C:0 T:1 74147 XUs and 42156 XVs -> 4024 XUVs
C:0 T:2 69511 XUs and 62763 XVs -> 2549 XUVs
C:0 T:3 37786 XUs and 29011 XVs -> 192 XUVs
C:0 T:4 17573 XUs and 17486 XVs -> 178 XUVs
C:0 T:5 97226 XUs and 93592 XVs -> 8569 XUVs
C:0 T:6 144713 XUs and 136922 XVs -> 8885 XUVs
C:0 T:7 54666 XUs and 48040 XVs -> 971 XUVs
C:0 T:8 6703 XUs and 12383 XVs -> 147 XUVs
C:0 T:9 66563 XUs and 173607 XVs -> 11583 XUVs
C:0 T:10 29025 XUs and 21468 XVs -> 4732 XUVs
C:0 T:11 148 XUs and 95 XVs -> 22 XUVs
42070 XUVs total
13694 collection wire objects
38724 potential space points
Neighbour search...
2832402 tests to find 1448340 neighbours
Iterating with no regularization...
Begin: 8.42031e+07
0 5.95869e+07
1 5.80771e+07
2 5.78494e+07
3 5.77716e+07
4 5.77352e+07
Now with regularization...
Begin: 4.64312e+07
0 4.63869e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411012849
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411012849
fLifetime: 30000 [us]
09-Dec-2025 21:35:12 GMT  Closed output file "reco_stage1.root"
09-Dec-2025 21:35:12 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          362.32        522.941       864.494       506.425       128.623        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00203371    0.00680881     0.0142842    0.00677095    0.00375864       10     
reco:ophit:OpHitFinder                             0.272237      0.398226      0.538914      0.400683      0.0893717       10     
reco:opflash:OpFlashFinder                         0.147479       0.23081      0.309172      0.214495      0.0518901       10     
reco:opslicer:OpSlicer                              0.52895      0.838467       1.13877      0.855274      0.187839        10     
reco:caldata:DataPrepModule                         6.72271       8.07354       8.98876       8.21392      0.727525        10     
reco:wclsdatasp:WireCellToolkit                     39.9147       54.9071       73.4662       55.9669       9.00543        10     
reco:gaushit:GausHitFinder                          1.06675       1.82204       2.8318        1.87745       0.50114        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000630864   0.00126192    0.00293201    0.000898276   0.000845444      10     
reco:reco3d:SpacePointSolver                        13.7862       23.1634       50.9352       19.0137       10.6034        10     
reco:hitpdune:DisambigFromSpacePoints               1.7702        2.46392       4.31244       2.06097      0.769614        10     
reco:pandora:StandardPandora                        27.7202       55.9324       136.444       49.463        27.9934        10     
reco:pandoraWriter:StandardPandora                 0.474069      0.718745       1.22163      0.704567       0.19904        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.65243       9.60003       21.3932       8.4592        4.45011        10     
reco:pandoraShower:LArPandoraShowerCreation         3.79396       10.2203       27.1601       9.08079       6.20125        10     
reco:pandoracalo:Calorimetry                        2.12071       4.9253        12.0948       4.39621       2.54614        10     
reco:pandoracalonosce:Calorimetry                   1.55163       4.73908       11.9623       4.05381       2.68052        10     
reco:pandorapid:Chi2ParticleID                    0.00586394    0.00956751     0.0150292    0.00927921     0.0025689       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0468036     0.0853664     0.138753      0.0807772     0.0252254       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00541383    0.00852483     0.0112897    0.00850858    0.00192523       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.43155       10.3539       29.4822       8.43103       6.76074        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.07212       9.84784       30.0811       7.93242       7.12218        10     
reco:emtrkmichelid:EmTrackMichelId                  192.311       249.598       331.391       247.972       38.2902        10     
reco:crttag:SingleCRTMatchingProducer              0.0174877     0.0261868     0.0365791     0.0253385     0.0056303       10     
reco:crtreco:TwoCRTMatchingProducer               0.00989015     0.0182934     0.0348071     0.0159269    0.00690432       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.747891       1.04205       1.30064       1.08029      0.152015        10     
reco:pandora2Track:LArPandoraTrackCreation          13.718        23.1319       34.7953       22.9652       6.24569        10     
reco:pandora2calo:Calorimetry                       6.14799       11.8695       20.175        11.5679       4.39187        10     
reco:pandora2calonosce:Calorimetry                  6.68649       11.3689       19.1751       10.5413       3.69035        10     
reco:pandora2pid:Chi2ParticleID                    0.0121504     0.0216659     0.030532      0.021511     0.00445713       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0639891     0.105201      0.140839      0.104014      0.0230979       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0106702     0.0189277     0.0260632     0.0196407     0.0051925       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.92471       7.11885       12.6926       6.4932        2.42759        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.49354       8.72183       15.3977       8.07029       3.02411        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.19158       7.61712       12.2725       6.37252       2.5239         10     
[art]:TriggerResults:TriggerResultInserter        8.0811e-05    0.000120308   0.000234931   0.000105177   4.49065e-05      10     
end_path:out1:RootOutput                          1.2313e-05    1.77765e-05    3.169e-05    1.64105e-05   5.53311e-06      10     
end_path:out1:RootOutput(write)                     2.68171       3.66654       5.10682       3.5962       0.731865        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5208.08 MB
  Peak resident set size usage (VmHWM): 4359.28 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.40 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.73 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10962024
-rw-r--r-- 1 dune01 dune01 2552699960 Dec  9 20:05 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2427242117 Dec  9 20:02 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2242891899 Dec  9 21:35 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2009887473 Dec  9 17:38 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1915808197 Dec  9 17:23 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   75950840 Dec  9 21:28 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242751 Dec  9 21:35 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  9 21:35 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  9 21:35 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  9 21:35 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  9 20:05 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  9 17:38 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  9 21:35 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  9 17:15 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  9 21:35 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  9 20:18 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  9 20:05 debugprod.log
justIN time: 2026-02-07 14:58:10 UTC       justIN version: 01.06.00