justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 240906.0@dunegpschedd02.fnal.gov

Jobsub ID240906.0@dunegpschedd02.fnal.gov
Workflow ID9897
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-10 13:29:01
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-10 18:33:27
From worker nodeHostnamewn1208241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-10 13:29:27
Input filesmonte-carlo-009897-000001
JobscriptExit code0
Real time5h (18145s)
CPU time4h (17483s = 96%)
Max RSS bytes3873005568 (3693 MiB)
Outputting started2025-11-10 18:31:53
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09897/1/001/reco_stage1.root
Finished2025-11-10 18:33:27
Saved logsjustin-logs:240906.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 104415 traces, 0 summary
Retagger: tagging trace set: gauss with 76795 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76795 traces tagged "gauss"
FrameSaver: q=2.78373e+06 n=1340221 tag=gauss
wclsFrameSaver: saving 104415 traces tagged "wiener"
FrameSaver: q=2.96253e+06 n=1180831 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 20242 XUs and 20849 XVs -> 145 XUVs
C:0 T:1 34971 XUs and 30723 XVs -> 2331 XUVs
C:0 T:2 82852 XUs and 102835 XVs -> 8758 XUVs
C:0 T:3 43514 XUs and 46139 XVs -> 183 XUVs
C:0 T:4 7967 XUs and 7874 XVs -> 93 XUVs
C:0 T:5 71754 XUs and 99188 XVs -> 32928 XUVs
C:0 T:6 76470 XUs and 104686 XVs -> 11594 XUVs
C:0 T:7 24392 XUs and 29041 XVs -> 460 XUVs
C:0 T:8 4195 XUs and 3704 XVs -> 34 XUVs
C:0 T:9 21011 XUs and 25456 XVs -> 2116 XUVs
C:0 T:10 75369 XUs and 91586 XVs -> 24538 XUVs
C:0 T:11 568 XUs and 483 XVs -> 54 XUVs
83234 XUVs total
8353 collection wire objects
79644 potential space points
Neighbour search...
23632864 tests to find 9858552 neighbours
Iterating with no regularization...
Begin: 5.17725e+07
0 4.33043e+07
1 4.27818e+07
2 4.26952e+07
3 4.26626e+07
Now with regularization...
Begin: 3.48053e+07
0 3.47243e+07
1 3.46594e+07
2 3.46031e+07
3 3.4555e+07
4 3.45141e+07
5 3.44788e+07
6 3.44487e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410403802
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410403802
fLifetime: 30000 [us]
11-Nov-2025 00:31:50 GMT  Closed output file "reco_stage1.root"
11-Nov-2025 00:31:50 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          445.902       594.022       802.264       546.44        130.529        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00189632    0.00461013    0.00659214    0.00506471    0.00152645       10     
reco:ophit:OpHitFinder                             0.284359      0.392653      0.474544      0.388817      0.064889        10     
reco:opflash:OpFlashFinder                         0.140171      0.212022      0.284864      0.208622      0.0408399       10     
reco:opslicer:OpSlicer                             0.624774       1.02347       1.50066       1.00424      0.254783        10     
reco:caldata:DataPrepModule                         6.62127       8.82064       9.92814       9.09086      0.957245        10     
reco:wclsdatasp:WireCellToolkit                     51.2331       65.1433       76.686        66.7738       9.66995        10     
reco:gaushit:GausHitFinder                          1.17316       1.76109       2.43516       1.68389      0.454778        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000603276   0.000970274    0.0013774    0.000999092   0.000256595      10     
reco:reco3d:SpacePointSolver                        12.0595       22.0081       37.7516       20.9691       8.38264        10     
reco:hitpdune:DisambigFromSpacePoints               1.46803       2.17803       3.3084        1.99922      0.651164        10     
reco:pandora:StandardPandora                        28.1217       45.9318       72.4245       42.4372       15.9986        10     
reco:pandoraWriter:StandardPandora                 0.500417      0.741839       1.05613      0.710196      0.168672        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.42702       7.30672       10.1257       6.74194       1.99648        10     
reco:pandoraShower:LArPandoraShowerCreation         3.07335       7.41475       13.1497       6.62358       3.13968        10     
reco:pandoracalo:Calorimetry                        1.77767       3.95862       6.20711       3.60749       1.31673        10     
reco:pandoracalonosce:Calorimetry                   1.96794       3.82462       7.26783       3.12159       1.58764        10     
reco:pandorapid:Chi2ParticleID                    0.00548428     0.0096525     0.0117056     0.0100342    0.00175948       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0548346     0.0895789     0.121034      0.0848399     0.0190457       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00539712    0.00936723     0.0115182    0.00940478    0.00166357       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.81067       8.52737       18.5757       6.46002       5.07806        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.99748       8.21398       16.224        6.7525        4.24581        10     
reco:emtrkmichelid:EmTrackMichelId                  261.007       331.374       423.813       308.621       58.9989        10     
reco:crttag:SingleCRTMatchingProducer              0.014981      0.0216073     0.0325061     0.0209768     0.0049349       10     
reco:crtreco:TwoCRTMatchingProducer                0.0138232     0.0191764     0.0285747     0.0187716    0.00386508       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.00946       1.29653       1.60798       1.30311      0.173655        10     
reco:pandora2Track:LArPandoraTrackCreation          11.339        22.0007       35.6269       20.2602       8.30155        10     
reco:pandora2calo:Calorimetry                       4.67066       11.3734       19.4002       11.0447       5.3837         10     
reco:pandora2calonosce:Calorimetry                  4.79321       10.2569       18.2799       8.45136       4.82615        10     
reco:pandora2pid:Chi2ParticleID                    0.0132331     0.0191102     0.0276088     0.0184411     0.0043643       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0599319     0.0991531     0.133611      0.0951466     0.0205522       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0130396     0.0202346     0.0273273     0.0204042    0.00410957       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.28195       7.58608       11.7363       6.67876       2.90541        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.85719       9.47465       18.3121       8.30072       4.9927         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.68379       8.40077       15.8666       7.25923       3.96831        10     
[art]:TriggerResults:TriggerResultInserter         4.81e-05     6.67144e-05   0.000182353   5.33505e-05   3.88147e-05      10     
end_path:out1:RootOutput                           7.013e-06    8.9771e-06    1.7233e-05    8.1305e-06    2.8467e-06       10     
end_path:out1:RootOutput(write)                     3.26519       4.29635       5.27258       4.07702      0.725136        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4731.05 MB
  Peak resident set size usage (VmHWM): 3873.01 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.53 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9059784
-rw-r--r-- 1 dune01 dune01 2168926657 Nov 10 22:49 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2064350576 Nov 10 22:45 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1800758492 Nov 11 00:31 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1622199301 Nov 10 19:56 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1558425622 Nov 10 19:38 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   61925510 Nov 11 00:25 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242843 Nov 11 00:31 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 11 00:31 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 11 00:31 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 11 00:31 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 10 22:49 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 10 19:56 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 11 00:31 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 10 19:29 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 11 00:31 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 10 22:57 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 10 22:49 debugprod.log
justIN time: 2026-02-10 09:53:41 UTC       justIN version: 01.06.00