justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237823.0@dunegpschedd02.fnal.gov

Jobsub ID237823.0@dunegpschedd02.fnal.gov
Workflow ID9622
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-05 14:03:28
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-05 19:30:52
From worker nodeHostnamewn1904241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 14:04:38
Input filesmonte-carlo-009622-000001
JobscriptExit code0
Real time5h (19466s)
CPU time5h (18660s = 95%)
Max RSS bytes4500852736 (4292 MiB)
Outputting started2025-11-05 19:29:06
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09622/1/001/reco_stage1.root
Finished2025-11-05 19:30:52
Saved logsjustin-logs:237823.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 156585 traces, 0 summary
Retagger: tagging trace set: gauss with 118471 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 118471 traces tagged "gauss"
FrameSaver: q=3.95366e+06 n=2121916 tag=gauss
wclsFrameSaver: saving 156585 traces tagged "wiener"
FrameSaver: q=4.17927e+06 n=1869286 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33332 XUs and 40030 XVs -> 427 XUVs
C:0 T:1 80833 XUs and 90600 XVs -> 7305 XUVs
C:0 T:2 96113 XUs and 125104 XVs -> 5462 XUVs
C:0 T:3 49266 XUs and 58412 XVs -> 661 XUVs
C:0 T:4 14447 XUs and 12680 XVs -> 121 XUVs
C:0 T:5 64703 XUs and 45334 XVs -> 4950 XUVs
C:0 T:6 132323 XUs and 157179 XVs -> 6847 XUVs
C:0 T:7 47972 XUs and 54612 XVs -> 480 XUVs
C:0 T:8 11845 XUs and 9138 XVs -> 160 XUVs
C:0 T:9 88833 XUs and 80523 XVs -> 6761 XUVs
C:0 T:10 58599 XUs and 41640 XVs -> 5901 XUVs
C:0 T:11 111 XUs and 90 XVs -> 4 XUVs
39079 XUVs total
15340 collection wire objects
34489 potential space points
Neighbour search...
2078305 tests to find 1120476 neighbours
Iterating with no regularization...
Begin: 7.74669e+07
0 6.59475e+07
1 6.52006e+07
2 6.50925e+07
3 6.50584e+07
Now with regularization...
Begin: 5.4777e+07
0 5.47571e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410329346
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410329346
fLifetime: 30000 [us]
05-Nov-2025 19:29:02 GMT  Closed output file "reco_stage1.root"
05-Nov-2025 19:29:02 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          471.118       714.561       1137.64       691.089       188.164        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00143953    0.00849602     0.0129792    0.00908539    0.00297556       10     
reco:ophit:OpHitFinder                             0.339662      0.434329      0.622589      0.423401      0.0714522       10     
reco:opflash:OpFlashFinder                         0.196934      0.290081      0.434572      0.285329      0.0589548       10     
reco:opslicer:OpSlicer                             0.771588       1.05379       1.37337       1.03483      0.161673        10     
reco:caldata:DataPrepModule                         7.22062       7.63031       8.79601       7.53087      0.424607        10     
reco:wclsdatasp:WireCellToolkit                     44.8573       49.3712       58.8419       47.7261       4.50605        10     
reco:gaushit:GausHitFinder                          1.42435       2.11608       2.95308       2.06418      0.420645        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000769078   0.00112409    0.00170856    0.00108953    0.000296597      10     
reco:reco3d:SpacePointSolver                        13.6465       34.1938       59.8947       29.5062       13.8967        10     
reco:hitpdune:DisambigFromSpacePoints               1.89297       3.26536       5.39794       3.03505       1.00348        10     
reco:pandora:StandardPandora                        41.3578       97.7685       325.053       73.3616       79.1712        10     
reco:pandoraWriter:StandardPandora                 0.643946      0.853338       1.2085       0.818184      0.157945        10     
reco:pandoraTrack:LArPandoraTrackCreation           9.18804       12.7058       16.7068       12.2988       2.38266        10     
reco:pandoraShower:LArPandoraShowerCreation         9.29781       16.5274       24.0749       16.6722       4.31293        10     
reco:pandoracalo:Calorimetry                        5.58794       7.82623       10.0669       7.77031       1.46743        10     
reco:pandoracalonosce:Calorimetry                   5.49736       7.49378       9.74802       7.40428       1.42458        10     
reco:pandorapid:Chi2ParticleID                    0.00853036     0.0106258     0.0148837    0.00979728    0.00214065       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0768659     0.106615      0.159694      0.100272      0.0247857       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00810238    0.00957736     0.0130535    0.00932876    0.00130294       10     
reco:pandoraShowercalo:ShowerCalorimetry            10.4233       18.2765       25.4046       19.188        4.62138        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       9.91434       17.7714       27.4032       18.2665       4.90497        10     
reco:emtrkmichelid:EmTrackMichelId                  212.768       289.577       386.805       280.416       52.7227        10     
reco:crttag:SingleCRTMatchingProducer              0.0232679     0.0415827     0.0675118     0.0376288     0.0130963       10     
reco:crtreco:TwoCRTMatchingProducer                0.0175013     0.0269214     0.0360596     0.0271406    0.00515496       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.931682       1.28681       1.71639       1.24913      0.222525        10     
reco:pandora2Track:LArPandoraTrackCreation          25.4477       41.3656       63.9876       41.4281       10.8106        10     
reco:pandora2calo:Calorimetry                       14.7343       24.5912       38.6906       24.3955       6.35202        10     
reco:pandora2calonosce:Calorimetry                  14.2232       23.4203       35.1983       23.3593       6.39067        10     
reco:pandora2pid:Chi2ParticleID                    0.0190508     0.0256638     0.036017      0.0253376    0.00485458       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.095885       0.12298      0.159177      0.121074      0.0192225       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0184074      0.02287      0.0273403     0.0227749    0.00270098       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.06429       14.4526       20.3006       14.6131       3.52122        10     
reco:pandora2Showercalo:ShowerCalorimetry           10.0301       19.9368       28.4079       20.6635       5.26778        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      9.27765       17.0599       23.8619       17.2078       4.46258        10     
[art]:TriggerResults:TriggerResultInserter        6.6746e-05    0.000118457   0.000280508   9.4513e-05    5.84935e-05      10     
end_path:out1:RootOutput                          1.1442e-05    2.00599e-05   3.9805e-05    1.65465e-05   8.66874e-06      10     
end_path:out1:RootOutput(write)                     3.47704       4.62819       6.29125       4.49354      0.809686        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5348.23 MB
  Peak resident set size usage (VmHWM): 4500.85 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.36 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.74 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11488144
-rw-r--r-- 1 dune01 dune01 2655647813 Nov  5 17:26 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2516222721 Nov  5 17:23 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2411178920 Nov  5 19:29 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2085648630 Nov  5 14:33 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 2002684614 Nov  5 14:14 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   91846830 Nov  5 19:20 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243223 Nov  5 19:29 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  5 19:29 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  5 19:29 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  5 19:29 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  5 17:26 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  5 14:33 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  5 19:29 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  5 14:04 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  5 19:29 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  5 17:40 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  5 17:27 debugprod.log
justIN time: 2026-02-11 06:07:23 UTC       justIN version: 01.06.00