justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237156.1@dunegpschedd02.fnal.gov

Jobsub ID237156.1@dunegpschedd02.fnal.gov
Workflow ID9582
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-04 07:51:50
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-04 12:20:50
From worker nodeHostnamewn1905241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 07:52:21
Input filesmonte-carlo-009582-000001
JobscriptExit code0
Real time4h (16002s)
CPU time4h (15513s = 96%)
Max RSS bytes4556685312 (4345 MiB)
Outputting started2025-11-04 12:19:03
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09582/1/001/reco_stage1.root
Finished2025-11-04 12:20:50
Saved logsjustin-logs:237156.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 165093 traces, 0 summary
Retagger: tagging trace set: gauss with 124417 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 124417 traces tagged "gauss"
FrameSaver: q=4.33118e+06 n=2168905 tag=gauss
wclsFrameSaver: saving 165093 traces tagged "wiener"
FrameSaver: q=4.60669e+06 n=1907980 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 37902 XUs and 37722 XVs -> 327 XUVs
C:0 T:1 81453 XUs and 70411 XVs -> 5112 XUVs
C:0 T:2 59216 XUs and 44299 XVs -> 2359 XUVs
C:0 T:3 31886 XUs and 18736 XVs -> 132 XUVs
C:0 T:4 12686 XUs and 16001 XVs -> 160 XUVs
C:0 T:5 82926 XUs and 66190 XVs -> 6730 XUVs
C:0 T:6 200282 XUs and 220342 XVs -> 28072 XUVs
C:0 T:7 61193 XUs and 69028 XVs -> 1602 XUVs
C:0 T:8 6064 XUs and 8943 XVs -> 107 XUVs
C:0 T:9 35891 XUs and 62849 XVs -> 2644 XUVs
C:0 T:10 102148 XUs and 113051 XVs -> 10620 XUVs
C:0 T:11 755 XUs and 562 XVs -> 57 XUVs
57922 XUVs total
15379 collection wire objects
52916 potential space points
Neighbour search...
8642336 tests to find 3789844 neighbours
Iterating with no regularization...
Begin: 1.03866e+08
0 7.90335e+07
1 7.76213e+07
2 7.74465e+07
3 7.74012e+07
Now with regularization...
Begin: 6.50849e+07
0 6.50164e+07
1 6.49756e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410304002
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410304002
fLifetime: 30000 [us]
04-Nov-2025 12:19:01 GMT  Closed output file "reco_stage1.root"
04-Nov-2025 12:19:01 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          428.602       525.442       609.537       526.297       60.1275        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00122449    0.00555941    0.00828078    0.00586365    0.00212624       10     
reco:ophit:OpHitFinder                             0.308338      0.386452      0.466402      0.403392      0.0581378       10     
reco:opflash:OpFlashFinder                         0.211872      0.251671       0.27806      0.253346      0.0212607       10     
reco:opslicer:OpSlicer                             0.666604      0.889209       1.07915      0.933692      0.138141        10     
reco:caldata:DataPrepModule                         5.43804       7.17122       8.18402       7.39647      0.923475        10     
reco:wclsdatasp:WireCellToolkit                     34.9148       43.7669       53.2111       43.3255       4.97946        10     
reco:gaushit:GausHitFinder                          1.53353       1.83095       2.36405       1.76788       0.23332        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000567797   0.000958593   0.00167684    0.00081935    0.000382583      10     
reco:reco3d:SpacePointSolver                        9.51109       26.0647       41.7795       25.2869       8.84675        10     
reco:hitpdune:DisambigFromSpacePoints               1.66948       2.67037       3.55071       2.5228       0.532952        10     
reco:pandora:StandardPandora                        30.5032       55.1844       81.9952       54.2552       16.1142        10     
reco:pandoraWriter:StandardPandora                 0.575593      0.723395      0.933292      0.724482      0.0994929       10     
reco:pandoraTrack:LArPandoraTrackCreation           6.25607       8.82622       17.2469       7.80368       3.00598        10     
reco:pandoraShower:LArPandoraShowerCreation         5.95298       9.75658       17.7226       8.6373        3.54971        10     
reco:pandoracalo:Calorimetry                        3.55661       4.9568        9.63057       4.54095       1.67544        10     
reco:pandoracalonosce:Calorimetry                   3.10316       4.40487       8.07576       3.80362       1.42109        10     
reco:pandorapid:Chi2ParticleID                    0.00666434     0.0088371     0.0103422    0.00875933    0.00109823       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0593005     0.0820911     0.106779      0.0819657     0.0153313       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00536089    0.00812348     0.0101437    0.00834281    0.00130552       10     
reco:pandoraShowercalo:ShowerCalorimetry            6.00078       10.0151       16.0186       8.34509       3.17005        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.95189       9.70535       17.8708       8.40612       3.62915        10     
reco:emtrkmichelid:EmTrackMichelId                  227.778       255.134       283.355       260.457       19.0618        10     
reco:crttag:SingleCRTMatchingProducer              0.0163771     0.0242909     0.0361998     0.0236604    0.00542361       10     
reco:crtreco:TwoCRTMatchingProducer                0.0100713     0.0158516     0.0216087     0.0157564    0.00338253       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.827075       1.13003       1.32522       1.15485      0.141297        10     
reco:pandora2Track:LArPandoraTrackCreation          17.6603       23.2919       36.8756       21.7267       5.34704        10     
reco:pandora2calo:Calorimetry                       7.40055       13.0153       19.1835       12.214        3.87387        10     
reco:pandora2calonosce:Calorimetry                  6.06624       12.5819       17.4663       12.218        3.01697        10     
reco:pandora2pid:Chi2ParticleID                    0.0114107     0.0210147     0.0296131     0.0208537    0.00505272       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0543211     0.104369      0.153841       0.10302      0.0267609       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0111718     0.0194125     0.0247821     0.0202608    0.00408129       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.18788       8.15995       12.8321       7.88732       2.00907        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.75605       10.822        14.4577       10.7905       2.5374         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.00099       9.99988       14.5133       9.78329       2.35697        10     
[art]:TriggerResults:TriggerResultInserter        6.5794e-05    0.000111334   0.000243989   9.3821e-05    4.83869e-05      10     
end_path:out1:RootOutput                          1.4247e-05    1.94926e-05   3.4845e-05    1.7553e-05    5.59813e-06      10     
end_path:out1:RootOutput(write)                     3.00464       4.20732       4.95166       4.26788      0.555682        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5412.32 MB
  Peak resident set size usage (VmHWM): 4556.69 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.49 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11321388
-rw-r--r-- 1 dune01 dune01 2622578260 Nov  4 10:48 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2485496284 Nov  4 10:45 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2349514985 Nov  4 12:19 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2068841763 Nov  4 08:17 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1981870007 Nov  4 08:01 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   84171030 Nov  4 12:11 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243190 Nov  4 12:19 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  4 12:19 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  4 12:19 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  4 12:19 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  4 10:48 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  4 08:17 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  4 12:19 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  4 07:52 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  4 12:19 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  4 10:57 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  4 10:48 debugprod.log
justIN time: 2026-02-11 10:43:48 UTC       justIN version: 01.06.00