justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237154.1@dunegpschedd02.fnal.gov

Jobsub ID237154.1@dunegpschedd02.fnal.gov
Workflow ID9577
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-04 07:51:49
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-04 12:49:48
From worker nodeHostnamewn1905241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 07:52:24
Input filesmonte-carlo-009577-000001
JobscriptExit code0
Real time4h (17736s)
CPU time4h (17219s = 97%)
Max RSS bytes4624945152 (4410 MiB)
Outputting started2025-11-04 12:48:01
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09577/1/001/reco_stage1.root
Finished2025-11-04 12:49:48
Saved logsjustin-logs:237154.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 179558 traces, 0 summary
Retagger: tagging trace set: gauss with 135341 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 135341 traces tagged "gauss"
FrameSaver: q=4.9418e+06 n=2410771 tag=gauss
wclsFrameSaver: saving 179558 traces tagged "wiener"
FrameSaver: q=5.25497e+06 n=2139398 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 32897 XUs and 36345 XVs -> 502 XUVs
C:0 T:1 71814 XUs and 91687 XVs -> 9669 XUVs
C:0 T:2 91377 XUs and 140021 XVs -> 6623 XUVs
C:0 T:3 45962 XUs and 57847 XVs -> 513 XUVs
C:0 T:4 11531 XUs and 14819 XVs -> 266 XUVs
C:0 T:5 66951 XUs and 125948 XVs -> 18104 XUVs
C:0 T:6 104543 XUs and 165630 XVs -> 16430 XUVs
C:0 T:7 34762 XUs and 48002 XVs -> 694 XUVs
C:0 T:8 17841 XUs and 20719 XVs -> 489 XUVs
C:0 T:9 183928 XUs and 288800 XVs -> 24381 XUVs
C:0 T:10 63638 XUs and 49751 XVs -> 7871 XUVs
C:0 T:11 259 XUs and 228 XVs -> 20 XUVs
85562 XUVs total
16841 collection wire objects
78533 potential space points
Neighbour search...
14751023 tests to find 6674204 neighbours
Iterating with no regularization...
Begin: 1.27927e+08
0 1.07734e+08
1 1.06326e+08
2 1.06124e+08
3 1.0606e+08
Now with regularization...
Begin: 8.94433e+07
0 8.93401e+07
1 8.92647e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410304062
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410304062
fLifetime: 30000 [us]
04-Nov-2025 12:47:58 GMT  Closed output file "reco_stage1.root"
04-Nov-2025 12:47:58 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          367.804       579.525       743.561       603.145       102.873        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00111812    0.00582661     0.0124105    0.00499062    0.00305062       10     
reco:ophit:OpHitFinder                             0.346011      0.444887      0.521421      0.456994      0.0638984       10     
reco:opflash:OpFlashFinder                          0.20157      0.297268      0.383384      0.303531      0.0556134       10     
reco:opslicer:OpSlicer                             0.736453      0.989867       1.27901      0.991545       0.1695         10     
reco:caldata:DataPrepModule                         5.07224       7.13997       8.3982        7.22622      0.920185        10     
reco:wclsdatasp:WireCellToolkit                     38.1546       43.356        52.201        43.1614       3.54026        10     
reco:gaushit:GausHitFinder                          1.31745       1.81848       2.39042       1.74586      0.418275        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000526159   0.000746233   0.00119853    0.000731575   0.000187723      10     
reco:reco3d:SpacePointSolver                        10.9911       32.8401       70.4328       32.0096       14.6576        10     
reco:hitpdune:DisambigFromSpacePoints               1.66651       2.73539       3.74343       2.68826      0.675206        10     
reco:pandora:StandardPandora                        28.1223       68.0957       92.1329       73.1618       21.0645        10     
reco:pandoraWriter:StandardPandora                 0.531727      0.806685       1.00231      0.827134      0.137349        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.85809       9.05237       11.1914       9.56979       1.80779        10     
reco:pandoraShower:LArPandoraShowerCreation         4.39531       11.3334       16.1153       12.0949       3.62248        10     
reco:pandoracalo:Calorimetry                        3.53604       5.25563       6.9776        5.1409        1.14491        10     
reco:pandoracalonosce:Calorimetry                   2.88039       5.0295        6.2687        5.46461       1.07172        10     
reco:pandorapid:Chi2ParticleID                    0.00574164     0.0101103     0.0127601     0.0101317    0.00195089       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0727918     0.0971438     0.130195      0.0986047     0.0140966       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0081367     0.0101296     0.0115063     0.0103564    0.00108169       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.12283       11.8532       16.7036       12.4213       3.92125        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.0204        11.3526       16.0699       12.0491       3.36894        10     
reco:emtrkmichelid:EmTrackMichelId                  195.276       276.676       330.11        289.947       39.9689        10     
reco:crttag:SingleCRTMatchingProducer              0.0147003     0.023883      0.0356317     0.0243127     0.0065592       10     
reco:crtreco:TwoCRTMatchingProducer               0.00758286     0.0156579     0.0217778     0.0157488    0.00467341       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.895241       1.13378       1.40185       1.15002      0.158779        10     
reco:pandora2Track:LArPandoraTrackCreation          13.7556       26.8265       40.8876       25.7263       7.99166        10     
reco:pandora2calo:Calorimetry                       8.3284        13.9564       20.6983       13.334        3.60785        10     
reco:pandora2calonosce:Calorimetry                  8.08583       14.1472       21.3143       14.2818       4.11831        10     
reco:pandora2pid:Chi2ParticleID                    0.0183594     0.0268973     0.0318201     0.0284952    0.00410555       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0922385     0.130796      0.190552      0.133879      0.0257373       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.016713      0.0263543     0.0363531     0.0265834    0.00494503       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.38917       9.13007       11.8006       9.74851       2.22164        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.93177       11.075        14.312        11.4297       2.50688        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.82422       9.30184       12.1826       9.18396       2.10097        10     
[art]:TriggerResults:TriggerResultInserter        5.2459e-05    0.000106303   0.000163878   0.000105849   3.30505e-05      10     
end_path:out1:RootOutput                           9.298e-06    1.42418e-05   1.8875e-05    1.4833e-05    2.93569e-06      10     
end_path:out1:RootOutput(write)                     3.15002       4.34197       5.9526        4.04344      0.832978        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5446.39 MB
  Peak resident set size usage (VmHWM): 4601.44 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.39 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 12290220
-rw-r--r-- 1 dune01 dune01 2809675718 Nov  4 11:08 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2664772692 Nov  4 11:05 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2570724270 Nov  4 12:47 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2263190515 Nov  4 08:20 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 2183294804 Nov  4 08:02 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   92898320 Nov  4 12:40 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242993 Nov  4 12:48 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  4 12:47 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  4 12:47 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  4 12:47 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  4 11:08 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  4 08:20 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  4 12:47 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  4 07:52 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  4 12:48 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  4 11:19 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  4 11:08 debugprod.log
justIN time: 2026-02-11 12:00:15 UTC       justIN version: 01.06.00