justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269088.0@dunegpschedd01.fnal.gov

Jobsub ID269088.0@dunegpschedd01.fnal.gov
Workflow ID11262
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-18 16:44:59
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-12-18 21:04:44
From worker nodeHostnamewn2207300.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 16:45:46
Input filesmonte-carlo-011262-000001
JobscriptExit code0
Real time4h (15398s)
CPU time4h (14891s = 96%)
Max RSS bytes4275634176 (4077 MiB)
Outputting started2025-12-18 21:02:25
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11262/1/001/reco_stage1.root
Finished2025-12-18 21:04:44
Saved logsjustin-logs:269088.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 131533 traces, 0 summary
Retagger: tagging trace set: gauss with 100431 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 100431 traces tagged "gauss"
FrameSaver: q=3.53307e+06 n=1750860 tag=gauss
wclsFrameSaver: saving 131533 traces tagged "wiener"
FrameSaver: q=3.73656e+06 n=1552916 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 32717 XUs and 40658 XVs -> 492 XUVs
C:0 T:1 65798 XUs and 131064 XVs -> 6677 XUVs
C:0 T:2 137314 XUs and 118631 XVs -> 5721 XUVs
C:0 T:3 55731 XUs and 47205 XVs -> 424 XUVs
C:0 T:4 8430 XUs and 10209 XVs -> 107 XUVs
C:0 T:5 24470 XUs and 25467 XVs -> 2567 XUVs
C:0 T:6 70425 XUs and 97462 XVs -> 3142 XUVs
C:0 T:7 28987 XUs and 41517 XVs -> 224 XUVs
C:0 T:8 9236 XUs and 9883 XVs -> 91 XUVs
C:0 T:9 54067 XUs and 71905 XVs -> 4549 XUVs
C:0 T:10 25066 XUs and 16416 XVs -> 2062 XUVs
C:0 T:11 223 XUs and 131 XVs -> 24 XUVs
26080 XUVs total
10688 collection wire objects
22826 potential space points
Neighbour search...
825962 tests to find 433744 neighbours
Iterating with no regularization...
Begin: 6.73783e+07
0 5.42434e+07
1 5.34321e+07
2 5.33296e+07
3 5.33036e+07
Now with regularization...
Begin: 4.38629e+07
0 4.38521e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411196485
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411196485
fLifetime: 30000 [us]
18-Dec-2025 21:02:21 GMT  Closed output file "reco_stage1.root"
18-Dec-2025 21:02:21 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          269.746       485.735       701.149       476.585       129.186        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00154093    0.00515855    0.00857558     0.0049801    0.00222825       10     
reco:ophit:OpHitFinder                             0.322361      0.419791      0.482546      0.428347      0.0470069       10     
reco:opflash:OpFlashFinder                          0.17725      0.229794       0.28013      0.233738      0.038803        10     
reco:opslicer:OpSlicer                             0.692503      0.935378       1.21226      0.937412      0.156501        10     
reco:caldata:DataPrepModule                         7.15064       7.70243       8.17076       7.67899      0.330025        10     
reco:wclsdatasp:WireCellToolkit                     35.9228       49.6675       65.2811       49.9092       7.34294        10     
reco:gaushit:GausHitFinder                          0.83361       1.80734       2.45983       1.90645      0.419487        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000530637   0.000830307   0.00145598    0.000784415   0.00027101       10     
reco:reco3d:SpacePointSolver                        10.1597       27.4547       48.7498       28.0586       12.0614        10     
reco:hitpdune:DisambigFromSpacePoints               1.23619       2.35993       3.15972       2.43959      0.647096        10     
reco:pandora:StandardPandora                        16.8581       49.1843       82.3529       45.5335       19.6781        10     
reco:pandoraWriter:StandardPandora                  0.27701       0.70342      0.986021      0.717935      0.203733        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.70232       6.28248       12.6424       4.97607       2.97666        10     
reco:pandoraShower:LArPandoraShowerCreation         2.43188       6.18039        10.47        5.53714       2.97167        10     
reco:pandoracalo:Calorimetry                        1.54691       3.32736       6.86434       3.27858       1.4772         10     
reco:pandoracalonosce:Calorimetry                   1.32314       3.08701       6.66542       2.8766        1.47229        10     
reco:pandorapid:Chi2ParticleID                    0.00343715    0.00836309     0.019676     0.00747626    0.00420073       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0311407     0.0768731     0.114051      0.073363      0.0246437       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00340738     0.0070584    0.00991454    0.00718433    0.00190407       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.6462        6.39984       12.5458       5.68541       3.34601        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.61012       6.12308       11.3961       5.73733       3.10564        10     
reco:emtrkmichelid:EmTrackMichelId                  157.272       253.707       373.124       249.083       61.4588        10     
reco:crttag:SingleCRTMatchingProducer              0.015338      0.0208633     0.0354015     0.0179719    0.00620914       10     
reco:crtreco:TwoCRTMatchingProducer                0.0079267     0.0116535     0.0184589     0.0106931    0.00346138       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.690496       1.05843       1.48445       1.04424      0.229269        10     
reco:pandora2Track:LArPandoraTrackCreation          6.34318       17.071        32.4119       16.1097       7.18519        10     
reco:pandora2calo:Calorimetry                       3.75484       8.68969       15.7008       8.37374       3.43313        10     
reco:pandora2calonosce:Calorimetry                  3.57074       8.25395       14.8223       8.03381       3.24325        10     
reco:pandora2pid:Chi2ParticleID                    0.0149267     0.0202991     0.0262927     0.0209105    0.00372714       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0746554     0.108537      0.145494      0.103193      0.0213906       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0149889     0.0202544     0.0272121     0.0203368    0.00354727       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.88259       6.27791       10.9657       5.47867       2.65346        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.3959        7.49941       13.6076       7.42504       2.93004        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.03676       6.83127       12.4123       6.68345       2.72784        10     
[art]:TriggerResults:TriggerResultInserter        7.7907e-05    0.000117157   0.000248387   9.63465e-05   4.81777e-05      10     
end_path:out1:RootOutput                          1.1932e-05    1.81071e-05   2.8644e-05    1.77485e-05   5.38791e-06      10     
end_path:out1:RootOutput(write)                     2.71181       4.00219        5.635        3.8919       0.906333        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5126.29 MB
  Peak resident set size usage (VmHWM): 4275.63 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.36 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.67 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10320108
-rw-r--r-- 1 dune01 dune01 2417631143 Dec 18 19:38 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2291268037 Dec 18 19:36 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2118601378 Dec 18 21:02 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1869963200 Dec 18 17:06 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1791617453 Dec 18 16:52 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   78082200 Dec 18 20:57 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242911 Dec 18 21:02 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec 18 21:02 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec 18 21:02 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec 18 21:02 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec 18 19:38 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 18 17:06 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 18 21:02 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec 18 16:45 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec 18 21:02 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec 18 19:48 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec 18 19:38 debugprod.log
justIN time: 2025-12-19 07:23:02 UTC       justIN version: 01.05.03