justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301990.1@dunegpschedd01.fnal.gov

Jobsub ID301990.1@dunegpschedd01.fnal.gov
Workflow ID12489
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-28 16:18:52
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-28 21:22:10
From worker nodeHostnamewn1206250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-28 16:20:29
Input filesmonte-carlo-012489-000001
JobscriptExit code0
Real time5h (18000s)
CPU time4h (17333s = 96%)
Max RSS bytes3964805120 (3781 MiB)
Outputting started2026-01-28 21:20:30
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12489/1/001/reco_stage1.root
Finished2026-01-28 21:22:10
Saved logsjustin-logs:301990.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 137164 traces, 0 summary
Retagger: tagging trace set: gauss with 111730 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 111730 traces tagged "gauss"
FrameSaver: q=3.65602e+06 n=2062740 tag=gauss
wclsFrameSaver: saving 137164 traces tagged "wiener"
FrameSaver: q=3.8426e+06 n=1864210 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 32879 XUs and 40276 XVs -> 355 XUVs
C:0 T:1 73536 XUs and 82732 XVs -> 5517 XUVs
C:0 T:2 96555 XUs and 151183 XVs -> 15770 XUVs
C:0 T:3 38808 XUs and 38076 XVs -> 344 XUVs
C:0 T:4 17959 XUs and 16117 XVs -> 141 XUVs
C:0 T:5 62234 XUs and 49843 XVs -> 3868 XUVs
C:0 T:6 173069 XUs and 167140 XVs -> 6749 XUVs
C:0 T:7 71633 XUs and 64357 XVs -> 796 XUVs
C:0 T:8 7198 XUs and 7136 XVs -> 79 XUVs
C:0 T:9 42184 XUs and 58447 XVs -> 2996 XUVs
C:0 T:10 38128 XUs and 24915 XVs -> 5186 XUVs
C:0 T:11 167 XUs and 107 XVs -> 27 XUVs
41828 XUVs total
12650 collection wire objects
38194 potential space points
Neighbour search...
6121314 tests to find 2458790 neighbours
Iterating with no regularization...
Begin: 6.67988e+07
0 5.65445e+07
1 5.58142e+07
2 5.57192e+07
3 5.56916e+07
Now with regularization...
Begin: 4.60427e+07
0 4.60204e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412021018
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412021018
fLifetime: 30000 [us]
28-Jan-2026 21:20:25 GMT  Closed output file "reco_stage1.root"
28-Jan-2026 21:20:25 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          448.987       639.307       891.111       621.43        149.524        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00157842    0.00408384    0.00548876    0.00444931    0.00122999       10     
reco:ophit:OpHitFinder                             0.356064      0.437991      0.488673      0.445603      0.0403989       10     
reco:opflash:OpFlashFinder                         0.151976      0.247273       0.30443      0.252675      0.0449911       10     
reco:opslicer:OpSlicer                             0.830987       1.14923       1.51952       1.17817      0.204948        10     
reco:caldata:DataPrepModule                         6.06552       8.52437       9.39941       8.79192      0.917246        10     
reco:wclsdatasp:WireCellToolkit                     48.9772       57.2877       66.8881       55.7522       5.92213        10     
reco:gaushit:GausHitFinder                          1.43053       1.96561       2.62546       1.90336      0.356719        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000666558   0.000909536   0.00125699    0.000908751   0.000204431      10     
reco:reco3d:SpacePointSolver                        12.3725       22.8847       36.5595       23.6472       7.32113        10     
reco:hitpdune:DisambigFromSpacePoints               1.3458        2.58785       4.35068       2.46479      0.898477        10     
reco:pandora:StandardPandora                        30.0848       57.1859       102.874       41.5536       28.7863        10     
reco:pandoraWriter:StandardPandora                 0.441398      0.726867      0.955966      0.755855      0.174632        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.82294       7.75241       11.789        7.54187       2.65578        10     
reco:pandoraShower:LArPandoraShowerCreation         4.33352       8.73548       16.8567       7.21353       4.48488        10     
reco:pandoracalo:Calorimetry                        2.30566       4.20049       6.40406       3.8421        1.66209        10     
reco:pandoracalonosce:Calorimetry                   2.11469       4.02791       6.44215       3.67502       1.52366        10     
reco:pandorapid:Chi2ParticleID                    0.00603874    0.00984996     0.0118338     0.0101563    0.00169627       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.064299      0.0970947     0.131009      0.0923121     0.0201544       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0060498    0.00945137     0.0114411    0.00966035     0.0015747       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.55952       9.43874       17.808        7.94878       4.96206        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.80135       8.84388        15.99        7.33392       4.26138        10     
reco:emtrkmichelid:EmTrackMichelId                  274.924       360.921       472.003       357.571       62.5393        10     
reco:crttag:SingleCRTMatchingProducer              0.0171979     0.0238753     0.0433933     0.0221361    0.00723066       10     
reco:crtreco:TwoCRTMatchingProducer                0.0115387     0.0176599     0.0291573     0.0167851    0.00438411       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.09933       1.47117       2.11621       1.48747      0.274007        10     
reco:pandora2Track:LArPandoraTrackCreation          11.6241       24.5331       41.0846       23.2923       9.28207        10     
reco:pandora2calo:Calorimetry                       5.32955       12.5417       21.2561       12.3651       4.86108        10     
reco:pandora2calonosce:Calorimetry                  4.28403       11.6257       17.5794       12.5181       4.67801        10     
reco:pandora2pid:Chi2ParticleID                    0.0132222     0.0239299     0.0333141     0.0250108    0.00623511       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0623662     0.127132      0.227056      0.125831      0.0447445       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0130415     0.0239514     0.0324312     0.0250237    0.00625659       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.36173       8.7815        13.368        9.11215       2.99424        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.72313       9.90956       15.5483       9.83124       3.86653        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.09728       8.41781       13.4121       7.85196       3.06037        10     
[art]:TriggerResults:TriggerResultInserter        3.4796e-05    5.70908e-05   0.00016938    4.67385e-05   3.78129e-05      10     
end_path:out1:RootOutput                           6.062e-06     7.999e-06    1.5009e-05     7.659e-06    2.47697e-06      10     
end_path:out1:RootOutput(write)                     3.43183       4.56964       6.4111        4.36276      0.937546        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4809.69 MB
  Peak resident set size usage (VmHWM): 3964.81 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.48 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9407200
-rw-r--r-- 1 dune01 dune01 2239957232 Jan 28 19:30 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2121234112 Jan 28 19:26 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1915890539 Jan 28 21:20 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1683523818 Jan 28 16:48 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1599312784 Jan 28 16:29 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   72425450 Jan 28 21:09 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242859 Jan 28 21:20 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 28 21:20 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 28 21:20 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 28 21:20 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 28 19:30 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 28 16:48 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 28 21:20 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 28 16:20 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 28 21:20 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 28 19:39 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 28 19:30 debugprod.log
justIN time: 2026-02-04 16:37:51 UTC       justIN version: 01.06.00