justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239163.1@dunegpschedd02.fnal.gov

Jobsub ID239163.1@dunegpschedd02.fnal.gov
Workflow ID9852
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-06 23:58:09
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-07 06:57:51
From worker nodeHostnamewn5917290.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-06 23:59:38
Input filesmonte-carlo-009852-000001
JobscriptExit code0
Real time6h (24357s)
CPU time6h (23601s = 96%)
Max RSS bytes4511338496 (4302 MiB)
Outputting started2025-11-07 06:45:36
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09852/1/001/reco_stage1.root
Finished2025-11-07 06:57:51
Saved logsjustin-logs:239163.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

epAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 184113 traces, 0 summary
Retagger: tagging trace set: gauss with 142214 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 142214 traces tagged "gauss"
FrameSaver: q=4.84714e+06 n=2511137 tag=gauss
wclsFrameSaver: saving 184113 traces tagged "wiener"
FrameSaver: q=5.10772e+06 n=2220268 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 47538 XUs and 49123 XVs -> 637 XUVs
C:0 T:1 117845 XUs and 118404 XVs -> 6721 XUVs
C:0 T:2 73751 XUs and 112574 XVs -> 3249 XUVs
C:0 T:3 35683 XUs and 47718 XVs -> 222 XUVs
C:0 T:4 15869 XUs and 17151 XVs -> 160 XUVs
C:0 T:5 69434 XUs and 58475 XVs -> 4213 XUVs
C:0 T:6 108299 XUs and 113215 XVs -> 4909 XUVs
C:0 T:7 45476 XUs and 39796 XVs -> 344 XUVs
C:0 T:8 15178 XUs and 13564 XVs -> 191 XUVs
C:0 T:9 121419 XUs and 127471 XVs -> 7331 XUVs
C:0 T:10 81152 XUs and 63682 XVs -> 7370 XUVs
C:0 T:11 405 XUs and 484 XVs -> 37 XUVs
35384 XUVs total
17421 collection wire objects
32330 potential space points
Neighbour search...
1096762 tests to find 574700 neighbours
Iterating with no regularization...
Begin: 1.04423e+08
0 8.68288e+07
1 8.57975e+07
2 8.5675e+07
3 8.56444e+07
Now with regularization...
Begin: 7.21155e+07
0 7.20881e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410352487
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410352487
fLifetime: 30000 [us]
07-Nov-2025 06:45:33 GMT  Closed output file "reco_stage1.root"
07-Nov-2025 06:45:33 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          624.986       841.21        990.58        847.274       100.595        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00217392     0.0052898    0.00906441    0.00524269    0.00200945       10     
reco:ophit:OpHitFinder                              0.39161      0.507836      0.597151      0.525883      0.0709209       10     
reco:opflash:OpFlashFinder                         0.311357      0.380095      0.477488      0.359147      0.0565819       10     
reco:opslicer:OpSlicer                              0.87032       1.40681       1.77147       1.46306      0.280556        10     
reco:caldata:DataPrepModule                         6.08749       8.35401       9.38949        8.691        1.20205        10     
reco:wclsdatasp:WireCellToolkit                     49.2336       61.8251       76.3896       59.2848       8.59707        10     
reco:gaushit:GausHitFinder                          1.42555       2.49639       3.13931       2.68634      0.519281        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000793476   0.00117124    0.00144627    0.00117073    0.000204098      10     
reco:reco3d:SpacePointSolver                        21.9308       33.3117       49.3945       34.304        8.57283        10     
reco:hitpdune:DisambigFromSpacePoints               2.97672       3.89991       4.83354       3.87047      0.564816        10     
reco:pandora:StandardPandora                        54.3358       88.0017       110.556       90.5321       15.0574        10     
reco:pandoraWriter:StandardPandora                 0.664709      0.865453       1.07757      0.868342      0.110669        10     
reco:pandoraTrack:LArPandoraTrackCreation           9.14275       11.7567       14.6548       11.7088       1.70456        10     
reco:pandoraShower:LArPandoraShowerCreation         9.19001       15.9776       19.5211       16.8675       2.84904        10     
reco:pandoracalo:Calorimetry                        3.89203       6.78492       9.12173       6.84927       1.33089        10     
reco:pandoracalonosce:Calorimetry                   3.73928       6.41091       8.2096        6.51696       1.15764        10     
reco:pandorapid:Chi2ParticleID                    0.00877272     0.011824      0.0144683     0.0114342    0.00153442       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0760934     0.114514      0.143006      0.113959      0.0215368       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00715038     0.0111636     0.0147924     0.0114933    0.00228429       10     
reco:pandoraShowercalo:ShowerCalorimetry            8.52445       15.5956       20.5825       15.7717       3.17645        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       8.1827        15.6779       19.9064       16.1068       3.14003        10     
reco:emtrkmichelid:EmTrackMichelId                   360.2        437.439       512.95        428.508       44.0994        10     
reco:crttag:SingleCRTMatchingProducer              0.0231076     0.0317001     0.0389414     0.0317975    0.00530156       10     
reco:crtreco:TwoCRTMatchingProducer                0.0198136     0.0259233     0.0300778     0.0262563     0.0030782       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.09656       1.71016       2.12854       1.73292      0.287099        10     
reco:pandora2Track:LArPandoraTrackCreation          22.8017       41.6093       57.0602       40.398        9.79871        10     
reco:pandora2calo:Calorimetry                       11.7397       19.7349       29.5794       19.9805       4.72356        10     
reco:pandora2calonosce:Calorimetry                  9.77846       19.1795       25.2379       21.2413       4.86565        10     
reco:pandora2pid:Chi2ParticleID                    0.0206254     0.0274756     0.0366264     0.0251222    0.00562939       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0942591      0.13194      0.180038      0.128407      0.0312707       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0176376     0.027297       0.03812       0.02541     0.00676114       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.18932       13.562        17.2334       13.5232       2.73791        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.03763       15.4984       21.1471       15.9699       3.95742        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.43538       13.0592       16.7788       13.8727       2.61787        10     
[art]:TriggerResults:TriggerResultInserter        3.3263e-05    5.34639e-05   0.000178857   4.1303e-05    4.21265e-05      10     
end_path:out1:RootOutput                           5.73e-06     7.7118e-06    1.6031e-05     7.039e-06    2.8317e-06       10     
end_path:out1:RootOutput(write)                     3.92853       5.53522       7.01324       5.56955      0.785631        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5347.82 MB
  Peak resident set size usage (VmHWM): 4511.34 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.35 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11882120
-rw-r--r-- 1 dune01 dune01 2716397111 Nov  7 04:21 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2568643624 Nov  7 04:17 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2479708336 Nov  7 06:45 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2196299709 Nov  7 00:42 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 2111828085 Nov  7 00:11 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   93784370 Nov  7 06:33 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243010 Nov  7 06:45 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  7 06:45 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  7 06:45 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  7 06:45 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  7 04:21 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  7 00:42 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  7 06:45 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  6 23:59 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  7 06:45 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  7 04:34 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  7 04:21 debugprod.log
justIN time: 2026-02-10 13:27:14 UTC       justIN version: 01.06.00