justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 251501.0@dunegpschedd01.fnal.gov

Jobsub ID251501.0@dunegpschedd01.fnal.gov
Workflow ID10209
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-15 08:06:32
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-15 12:01:49
From worker nodeHostnamewn2207300.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-15 08:07:13
Input filesmonte-carlo-010209-000001
JobscriptExit code0
Real time3h (13998s)
CPU time3h (13417s = 95%)
Max RSS bytes3914522624 (3733 MiB)
Outputting started2025-11-15 12:00:32
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10209/1/001/reco_stage1.root
Finished2025-11-15 12:01:49
Saved logsjustin-logs:251501.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 77088 traces, 0 summary
Retagger: tagging trace set: gauss with 58978 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 58978 traces tagged "gauss"
FrameSaver: q=1.99408e+06 n=1024795 tag=gauss
wclsFrameSaver: saving 77088 traces tagged "wiener"
FrameSaver: q=2.10456e+06 n=901553 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 18925 XUs and 17465 XVs -> 74 XUVs
C:0 T:1 34535 XUs and 30421 XVs -> 2463 XUVs
C:0 T:2 102178 XUs and 130202 XVs -> 4912 XUVs
C:0 T:3 47822 XUs and 54289 XVs -> 375 XUVs
C:0 T:4 6912 XUs and 5915 XVs -> 65 XUVs
C:0 T:5 14751 XUs and 10615 XVs -> 1194 XUVs
C:0 T:6 46857 XUs and 76267 XVs -> 4689 XUVs
C:0 T:7 15930 XUs and 23219 XVs -> 199 XUVs
C:0 T:8 2065 XUs and 3054 XVs -> 19 XUVs
C:0 T:9 8688 XUs and 14218 XVs -> 874 XUVs
C:0 T:10 11125 XUs and 13834 XVs -> 2289 XUVs
C:0 T:11 29 XUs and 15 XVs -> 5 XUVs
17158 XUVs total
7059 collection wire objects
15887 potential space points
Neighbour search...
815171 tests to find 453502 neighbours
Iterating with no regularization...
Begin: 3.80616e+07
0 3.3351e+07
1 3.29974e+07
2 3.29454e+07
3 3.29303e+07
Now with regularization...
Begin: 2.69974e+07
0 2.69829e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410525549
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410525549
fLifetime: 30000 [us]
15-Nov-2025 12:00:29 GMT  Closed output file "reco_stage1.root"
15-Nov-2025 12:00:29 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          335.009       534.83        790.878       509.379       144.704        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00178864     0.0102052     0.0361654    0.00807177    0.00923892       10     
reco:ophit:OpHitFinder                             0.290427       0.38685      0.457072      0.414432      0.063714        10     
reco:opflash:OpFlashFinder                         0.125101      0.211272      0.353422      0.204033      0.0680736       10     
reco:opslicer:OpSlicer                             0.630198       0.8905        1.14407       0.88086      0.172985        10     
reco:caldata:DataPrepModule                         7.55576       8.47993       9.39919       8.56663       0.53731        10     
reco:wclsdatasp:WireCellToolkit                     49.0457       57.7487       64.5117       57.8759       4.93494        10     
reco:gaushit:GausHitFinder                          1.2757        1.72263       2.44179       1.7442        0.35365        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000476606   0.00100802    0.00193102    0.000836398   0.000413044      10     
reco:reco3d:SpacePointSolver                        13.5605       24.6504       41.2524       22.7468       9.88995        10     
reco:hitpdune:DisambigFromSpacePoints               1.06561       2.16634       3.60344       2.14201      0.797396        10     
reco:pandora:StandardPandora                        17.9894       43.0851       79.8112       40.9602       17.3562        10     
reco:pandoraWriter:StandardPandora                 0.521705      0.747836       1.05023      0.761568      0.152324        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.60664       7.83344       14.1733       6.99004       2.95932        10     
reco:pandoraShower:LArPandoraShowerCreation         3.15736       9.81164       17.0892       9.01587       4.61229        10     
reco:pandoracalo:Calorimetry                        1.28847       4.83868       10.5737       4.12819       2.45965        10     
reco:pandoracalonosce:Calorimetry                   1.80792       4.37222       9.64724       3.96404       2.08249        10     
reco:pandorapid:Chi2ParticleID                    0.00620121     0.0108484     0.0222109    0.00915784    0.00454223       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0589153     0.101322      0.187338      0.090986      0.0370077       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00522661    0.00805044     0.0105975    0.00831039    0.00147576       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.4623        10.1264       21.1218       10.1849       5.64151        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.45888       8.69193       15.6062       8.09891       4.81216        10     
reco:emtrkmichelid:EmTrackMichelId                  186.106       256.574       348.529       250.111       51.7699        10     
reco:crttag:SingleCRTMatchingProducer              0.0107309     0.0392653     0.0964898     0.0354107     0.021635        10     
reco:crtreco:TwoCRTMatchingProducer               0.00794252     0.0251912     0.0422516     0.0252819    0.00952864       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.717262       1.10885       1.72164      0.917111      0.327275        10     
reco:pandora2Track:LArPandoraTrackCreation          10.0633       26.5764       46.6273       24.652        12.6961        10     
reco:pandora2calo:Calorimetry                       3.46188       14.1121       27.3071       12.2339       7.88619        10     
reco:pandora2calonosce:Calorimetry                  3.61481       14.1136       25.6389       12.6863       7.54196        10     
reco:pandora2pid:Chi2ParticleID                   0.00940177     0.0203042     0.0277036     0.0203784    0.00519325       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0406757     0.130537       0.22087      0.128575      0.0490165       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00783214     0.0235255     0.0581872     0.0191449     0.0126225       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.9106        9.23491       18.4407       8.30331       4.70923        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.83627       12.2669       26.5086       10.4479       7.19642        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.2391        10.7345       22.2248       9.1269        6.45758        10     
[art]:TriggerResults:TriggerResultInserter        8.8526e-05    0.00015338    0.000222799   0.00015728    3.50968e-05      10     
end_path:out1:RootOutput                          1.4557e-05    2.41323e-05   4.0716e-05    2.2387e-05    7.56892e-06      10     
end_path:out1:RootOutput(write)                     1.78485       3.64036       5.13962       3.62362      0.979656        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4763.46 MB
  Peak resident set size usage (VmHWM): 3914.52 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.44 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.99 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9126132
-rw-r--r-- 1 dune01 dune01 2187107528 Nov 15 10:28 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2081661421 Nov 15 10:26 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1831507458 Nov 15 12:00 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1620098003 Nov 15 08:26 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1559710072 Nov 15 08:15 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   64444620 Nov 15 11:56 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242811 Nov 15 12:00 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 15 12:00 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 15 12:00 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 15 12:00 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 15 10:28 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 15 08:26 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 15 12:00 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 15 08:07 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 15 12:00 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 15 10:35 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 15 10:28 debugprod.log
justIN time: 2026-02-09 12:26:18 UTC       justIN version: 01.06.00