justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264374.1@dunegpschedd01.fnal.gov

Jobsub ID264374.1@dunegpschedd01.fnal.gov
Workflow ID11064
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-06 11:33:58
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-06 17:06:48
From worker nodeHostnamewn5917301.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-06 11:34:48
Input filesmonte-carlo-011064-000001
JobscriptExit code0
Real time5h (19814s)
CPU time5h (19030s = 96%)
Max RSS bytes4120195072 (3929 MiB)
Outputting started2025-12-06 17:05:03
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11064/1/001/reco_stage1.root
Finished2025-12-06 17:06:48
Saved logsjustin-logs:264374.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 123732 traces, 0 summary
Retagger: tagging trace set: gauss with 94886 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 94886 traces tagged "gauss"
FrameSaver: q=3.20081e+06 n=1692782 tag=gauss
wclsFrameSaver: saving 123732 traces tagged "wiener"
FrameSaver: q=3.37824e+06 n=1497129 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 41637 XUs and 31823 XVs -> 415 XUVs
C:0 T:1 79266 XUs and 66110 XVs -> 5370 XUVs
C:0 T:2 120774 XUs and 110696 XVs -> 10744 XUVs
C:0 T:3 45191 XUs and 32257 XVs -> 463 XUVs
C:0 T:4 6552 XUs and 10273 XVs -> 151 XUVs
C:0 T:5 26063 XUs and 29106 XVs -> 3156 XUVs
C:0 T:6 125328 XUs and 82772 XVs -> 6029 XUVs
C:0 T:7 44552 XUs and 29415 XVs -> 389 XUVs
C:0 T:8 6921 XUs and 8228 XVs -> 88 XUVs
C:0 T:9 40281 XUs and 59376 XVs -> 2820 XUVs
C:0 T:10 41417 XUs and 19146 XVs -> 4692 XUVs
C:0 T:11 100 XUs and 49 XVs -> 9 XUVs
34326 XUVs total
11453 collection wire objects
30896 potential space points
Neighbour search...
2808282 tests to find 1381338 neighbours
Iterating with no regularization...
Begin: 6.81543e+07
0 5.79036e+07
1 5.7175e+07
2 5.70623e+07
3 5.7024e+07
Now with regularization...
Begin: 4.85125e+07
0 4.84785e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410949877
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410949877
fLifetime: 30000 [us]
06-Dec-2025 17:05:01 GMT  Closed output file "reco_stage1.root"
06-Dec-2025 17:05:01 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          487.138       641.715       837.719       629.222       122.273        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00140214    0.00490766    0.00938883    0.00452396    0.00238143       10     
reco:ophit:OpHitFinder                             0.325753      0.422907       0.5139       0.423942      0.0511828       10     
reco:opflash:OpFlashFinder                         0.183283       0.24132      0.296616      0.250176      0.0406515       10     
reco:opslicer:OpSlicer                             0.600461       1.09635       1.41344       1.13485      0.223899        10     
reco:caldata:DataPrepModule                         6.84631       7.88529       9.04618       7.72274      0.685993        10     
reco:wclsdatasp:WireCellToolkit                     46.8375       55.1922       60.7406       56.6301       4.96545        10     
reco:gaushit:GausHitFinder                          1.31877       1.79729       2.62267       1.71257       0.36764        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00056157    0.000784777   0.00114422    0.000770909   0.000155992      10     
reco:reco3d:SpacePointSolver                        14.0718       33.1192       79.221        24.0214       21.2691        10     
reco:hitpdune:DisambigFromSpacePoints               2.03495       2.5756        3.73948       2.36415      0.539024        10     
reco:pandora:StandardPandora                        34.2619       57.8771       88.5459       53.563        17.4989        10     
reco:pandoraWriter:StandardPandora                 0.511039      0.736202      0.974424      0.729382      0.134895        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.62263       8.35676       11.9704       7.60742       2.19569        10     
reco:pandoraShower:LArPandoraShowerCreation         5.68492       8.8947        14.9026       7.87875       3.20474        10     
reco:pandoracalo:Calorimetry                        2.93117       5.09716       7.55914       4.62885       1.48935        10     
reco:pandoracalonosce:Calorimetry                    2.724        4.71873       7.53101       4.29038       1.56007        10     
reco:pandorapid:Chi2ParticleID                    0.00655332     0.0105764     0.0133701     0.0105927    0.00176502       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0591273     0.0979676     0.133705      0.0944223     0.0207792       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.006354      0.0101819     0.0138396    0.00973337    0.00185859       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.31913       9.08748       14.2795       8.97629       2.91651        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.30498       8.54327       13.2531       7.75128       2.65629        10     
reco:emtrkmichelid:EmTrackMichelId                  284.309       353.196       440.305       343.497       54.9778        10     
reco:crttag:SingleCRTMatchingProducer             0.00986785     0.024201      0.0340244     0.024745     0.00607593       10     
reco:crtreco:TwoCRTMatchingProducer               0.00914794     0.0168702     0.0224916     0.0168718    0.00369947       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.10735       1.46576       1.91893       1.42172      0.288727        10     
reco:pandora2Track:LArPandoraTrackCreation          16.8696       25.0707       42.168        24.3613       7.76428        10     
reco:pandora2calo:Calorimetry                       6.03533       11.5304       19.8003       10.4251       3.84609        10     
reco:pandora2calonosce:Calorimetry                  5.80247        10.85        20.6376       9.75374       4.28941        10     
reco:pandora2pid:Chi2ParticleID                    0.0138987     0.0216989     0.0344792     0.0196878    0.00630209       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0671832     0.102057      0.164036      0.0909275     0.0343588       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0141492     0.021329      0.0335716     0.0195064     0.0059927       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.73095       8.86591       15.3529       8.99734       2.71078        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.12116       10.3701       18.6159       10.5009       3.42535        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.84903       9.36092       16.0177       8.65491       3.08985        10     
[art]:TriggerResults:TriggerResultInserter        3.8212e-05    5.92227e-05   0.000156886   4.9037e-05    3.32176e-05      10     
end_path:out1:RootOutput                           6.622e-06    8.1202e-06    1.3184e-05    7.7595e-06    1.82491e-06      10     
end_path:out1:RootOutput(write)                     3.56553       4.85393       6.39508       4.92935      0.883495        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4968.21 MB
  Peak resident set size usage (VmHWM): 4120.2 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.98 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9955344
-rw-r--r-- 1 dune01 dune01 2346539800 Dec  6 15:14 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2230291200 Dec  6 15:10 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2027973830 Dec  6 17:05 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1803780080 Dec  6 12:08 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1711175630 Dec  6 11:44 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73886470 Dec  6 16:56 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242953 Dec  6 17:05 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  6 17:05 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  6 17:05 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  6 17:05 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  6 15:14 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  6 12:08 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  6 17:05 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  6 11:34 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  6 17:05 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  6 15:23 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  6 15:14 debugprod.log
justIN time: 2026-02-07 17:42:24 UTC       justIN version: 01.06.00