justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 254068.1@dunegpschedd01.fnal.gov

Jobsub ID254068.1@dunegpschedd01.fnal.gov
Workflow ID10376
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-19 09:33:45
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-19 13:24:25
From worker nodeHostnamewn2204241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-19 09:34:58
Input filesmonte-carlo-010376-000001
JobscriptExit code0
Real time3h (13680s)
CPU time3h (13253s = 96%)
Max RSS bytes4008296448 (3822 MiB)
Outputting started2025-11-19 13:22:59
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10376/1/001/reco_stage1.root
Finished2025-11-19 13:24:25
Saved logsjustin-logs:254068.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

c
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 141110 traces, 0 summary
Retagger: tagging trace set: gauss with 103722 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 103722 traces tagged "gauss"
FrameSaver: q=3.69616e+06 n=1793291 tag=gauss
wclsFrameSaver: saving 141110 traces tagged "wiener"
FrameSaver: q=3.95691e+06 n=1566950 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 27264 XUs and 21234 XVs -> 618 XUVs
C:0 T:1 59762 XUs and 54825 XVs -> 5348 XUVs
C:0 T:2 182288 XUs and 278792 XVs -> 15728 XUVs
C:0 T:3 63588 XUs and 87811 XVs -> 1300 XUVs
C:0 T:4 9720 XUs and 11637 XVs -> 64 XUVs
C:0 T:5 30618 XUs and 27884 XVs -> 2682 XUVs
C:0 T:6 148891 XUs and 186855 XVs -> 15907 XUVs
C:0 T:7 50744 XUs and 53461 XVs -> 688 XUVs
C:0 T:8 4606 XUs and 6338 XVs -> 31 XUVs
C:0 T:9 25542 XUs and 44050 XVs -> 2998 XUVs
C:0 T:10 34630 XUs and 19839 XVs -> 3727 XUVs
C:0 T:11 71 XUs and 18 XVs -> 9 XUVs
49100 XUVs total
14154 collection wire objects
43425 potential space points
Neighbour search...
4907021 tests to find 2228672 neighbours
Iterating with no regularization...
Begin: 1.25996e+08
0 9.45217e+07
1 9.23529e+07
2 9.20989e+07
3 9.20332e+07
Now with regularization...
Begin: 7.63263e+07
0 7.61493e+07
1 7.60963e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410607189
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410607189
fLifetime: 30000 [us]
19-Nov-2025 13:22:57 GMT  Closed output file "reco_stage1.root"
19-Nov-2025 13:22:57 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          335.296       494.191       711.431       490.222       115.975        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00209766    0.00503317    0.00803526    0.00482528    0.00178882       10     
reco:ophit:OpHitFinder                             0.204243      0.308891      0.399373      0.333796      0.0604963       10     
reco:opflash:OpFlashFinder                         0.0918571     0.168597      0.273841      0.160652      0.0456709       10     
reco:opslicer:OpSlicer                             0.330329      0.732008       1.07533      0.762455      0.224474        10     
reco:caldata:DataPrepModule                         5.32703       6.8218        8.61203       6.18608       1.28395        10     
reco:wclsdatasp:WireCellToolkit                     43.2451       48.182        53.4026       47.2779       3.7678         10     
reco:gaushit:GausHitFinder                         0.840544       1.47755       2.32141       1.36609      0.494694        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000516259   0.00171402    0.00400619    0.00128617    0.00107355       10     
reco:reco3d:SpacePointSolver                        9.47514       24.8636       48.5477       21.5615       12.7387        10     
reco:hitpdune:DisambigFromSpacePoints               0.9064        2.0302        3.49222       1.80068      0.790049        10     
reco:pandora:StandardPandora                        16.0609       44.5055       76.8653       45.2565       19.0002        10     
reco:pandoraWriter:StandardPandora                 0.432351       0.66345      0.867742      0.642264      0.132725        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.96665       9.94719       17.9627       8.96874       4.41846        10     
reco:pandoraShower:LArPandoraShowerCreation         3.41035       12.485        24.3847       12.848        6.44289        10     
reco:pandoracalo:Calorimetry                        1.44684       5.47026       11.2189       4.96052       3.04119        10     
reco:pandoracalonosce:Calorimetry                   1.8035        5.55721       11.1395       5.02753       3.06528        10     
reco:pandorapid:Chi2ParticleID                    0.00370641    0.00809961     0.0161848    0.00711034    0.00359468       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0357549     0.0709965     0.111871      0.079093      0.0268909       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0037778    0.00734414     0.0147052    0.00719868    0.00314921       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.57721       12.5917       22.4859       12.3445       6.66891        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.27574       9.96915       19.7725       9.85133       5.12926        10     
reco:emtrkmichelid:EmTrackMichelId                  184.138       224.081       292.017       219.093       33.6958        10     
reco:crttag:SingleCRTMatchingProducer              0.0235166     0.0370987     0.0734995     0.0330362     0.0149011       10     
reco:crtreco:TwoCRTMatchingProducer                0.0134557     0.0253274     0.0432411     0.0248251    0.00809107       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.875028       1.19843       2.13157       1.06768      0.359098        10     
reco:pandora2Track:LArPandoraTrackCreation          16.3985       28.0869       49.2598       25.4442       10.5533        10     
reco:pandora2calo:Calorimetry                       4.24689       12.2774       20.8475       11.9591        5.74          10     
reco:pandora2calonosce:Calorimetry                  3.98388       11.4424       18.3652       11.0309       4.58259        10     
reco:pandora2pid:Chi2ParticleID                   0.00839311     0.0133104     0.0220198     0.0121246    0.00450358       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0443434     0.0710384     0.110815      0.0665647     0.0202808       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00839676     0.0134504     0.0213525     0.0119683    0.00445302       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.00791       8.97467       14.6184       7.87969       3.23713        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.21853       9.88456       15.4783       9.6387        3.1212         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.18638       8.58443       15.5763       8.08362       3.38963        10     
[art]:TriggerResults:TriggerResultInserter        5.5645e-05    9.94719e-05   0.000287359   7.47705e-05   6.61862e-05      10     
end_path:out1:RootOutput                          1.2764e-05    1.6523e-05    2.5347e-05    1.55745e-05   3.52199e-06      10     
end_path:out1:RootOutput(write)                     2.87061       3.34065       5.24422       3.09361      0.672197        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4838.28 MB
  Peak resident set size usage (VmHWM): 4008.3 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.04 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9144256
-rw-r--r-- 1 dune01 dune01 2186265829 Nov 19 11:58 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2078672498 Nov 19 11:55 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1856486169 Nov 19 13:22 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1618860602 Nov 19 09:57 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1551304696 Nov 19 09:42 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   71498260 Nov 19 13:15 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242883 Nov 19 13:22 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 19 13:22 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 19 13:22 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 19 13:22 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 19 11:58 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 19 09:57 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 19 13:22 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 19 09:34 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 19 13:22 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 19 12:04 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 19 11:58 debugprod.log
justIN time: 2026-02-09 05:00:02 UTC       justIN version: 01.06.00