justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 255877.1@dunegpschedd01.fnal.gov

Jobsub ID255877.1@dunegpschedd01.fnal.gov
Workflow ID10561
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-23 23:09:34
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-24 04:29:14
From worker nodeHostnamewn-la-02.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-23 23:10:34
Input filesmonte-carlo-010561-000001
JobscriptExit code0
Real time4h (17397s)
CPU time4h (17266s = 99%)
Max RSS bytes4421128192 (4216 MiB)
Outputting started2025-11-24 04:00:32
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10561/1/001/reco_stage1.root
Finished2025-11-24 04:29:14
Saved logsjustin-logs:255877.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 114682 traces, 0 summary
Retagger: tagging trace set: gauss with 91464 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 91464 traces tagged "gauss"
FrameSaver: q=3.23305e+06 n=1707465 tag=gauss
wclsFrameSaver: saving 114682 traces tagged "wiener"
FrameSaver: q=3.39453e+06 n=1546541 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 27008 XUs and 23184 XVs -> 177 XUVs
C:0 T:1 40384 XUs and 34562 XVs -> 2014 XUVs
C:0 T:2 50735 XUs and 48969 XVs -> 1905 XUVs
C:0 T:3 29215 XUs and 20881 XVs -> 163 XUVs
C:0 T:4 11463 XUs and 10441 XVs -> 149 XUVs
C:0 T:5 37260 XUs and 27434 XVs -> 3028 XUVs
C:0 T:6 67331 XUs and 68601 XVs -> 2918 XUVs
C:0 T:7 28043 XUs and 28322 XVs -> 268 XUVs
C:0 T:8 13461 XUs and 13112 XVs -> 286 XUVs
C:0 T:9 88306 XUs and 104006 XVs -> 5794 XUVs
C:0 T:10 50150 XUs and 57804 XVs -> 5070 XUVs
C:0 T:11 100 XUs and 104 XVs -> 12 XUVs
21784 XUVs total
10160 collection wire objects
20035 potential space points
Neighbour search...
553631 tests to find 303110 neighbours
Iterating with no regularization...
Begin: 7.13641e+07
0 6.46345e+07
1 6.4078e+07
2 6.40137e+07
3 6.39994e+07
Now with regularization...
Begin: 5.44731e+07
0 5.44773e+07
Warning: metric increased
run: 1 ; subrun: 0 ; event: 10
evttime: 410698064
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410698064
fLifetime: 30000 [us]
24-Nov-2025 05:00:30 CET  Closed output file "reco_stage1.root"
24-Nov-2025 05:00:30 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          337.634       482.032       584.521       474.725        85.36         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00115417    0.00611097     0.0099974     0.0061868    0.00285939       10     
reco:ophit:OpHitFinder                             0.396356       0.43303       0.50099       0.42468      0.0353465       10     
reco:opflash:OpFlashFinder                         0.233135      0.288568      0.371427       0.28655      0.0360373       10     
reco:opslicer:OpSlicer                             0.766722      0.918304       1.08101       0.90274      0.0950143       10     
reco:caldata:DataPrepModule                         6.35877       6.58007       7.25062       6.47309      0.283084        10     
reco:wclsdatasp:WireCellToolkit                     46.4875       65.7974       72.7015       71.1942       9.72425        10     
reco:gaushit:GausHitFinder                          1.18561       1.51109       1.77669       1.53058      0.176032        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000393368   0.000659689   0.00103927    0.000577739   0.000219314      10     
reco:reco3d:SpacePointSolver                        7.32982       22.9546       37.4047       22.5294       9.92846        10     
reco:hitpdune:DisambigFromSpacePoints               1.15746       1.9678        2.55826       1.88252      0.456903        10     
reco:pandora:StandardPandora                         21.76        41.5331       63.3544       38.5152       13.0494        10     
reco:pandoraWriter:StandardPandora                 0.389892      0.528984      0.647989       0.5466       0.0744384       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.2511        6.97056       10.7807       7.03898       1.83574        10     
reco:pandoraShower:LArPandoraShowerCreation         3.39549       8.64641       14.3663       9.02232       3.10034        10     
reco:pandoracalo:Calorimetry                        2.10764       3.92313       5.97102       3.98258       1.13244        10     
reco:pandoracalonosce:Calorimetry                   2.02149       3.69671       5.73103       3.72329       1.09219        10     
reco:pandorapid:Chi2ParticleID                    0.00547893     0.0066715    0.00798633    0.00641704    0.000766859      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0477562     0.0601886     0.0697063     0.0615453     0.0066291       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00524396     0.0063035    0.00802986    0.00626642    0.000736183      10     
reco:pandoraShowercalo:ShowerCalorimetry            3.50534       8.97553       14.7258       9.90036       3.0943         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.31963       8.58761       12.8712       9.40365       2.82609        10     
reco:emtrkmichelid:EmTrackMichelId                  172.766       222.101       258.153       224.866       29.7723        10     
reco:crttag:SingleCRTMatchingProducer              0.0102552     0.0205504     0.0299658     0.0218192    0.00663963       10     
reco:crtreco:TwoCRTMatchingProducer               0.00899369     0.0153208     0.0282874     0.0150425    0.00502857       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.911384       1.07086       1.20906       1.06279      0.108615        10     
reco:pandora2Track:LArPandoraTrackCreation          10.6084        22.59        35.1638       20.7946       7.78162        10     
reco:pandora2calo:Calorimetry                       4.60667       11.6321       18.3793       11.4523       3.9275         10     
reco:pandora2calonosce:Calorimetry                  4.31106       11.1011       17.7207       11.6501       3.87996        10     
reco:pandora2pid:Chi2ParticleID                    0.0115557     0.0155899     0.0202402     0.0152006    0.00253684       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0582783     0.0769604     0.0905425     0.0795404     0.0104108       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0115366     0.0153558     0.0199637     0.0150426    0.00240474       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.63829       8.09513       14.3998       8.54565       2.8629         10     
reco:pandora2Showercalo:ShowerCalorimetry           4.08396       9.7319        14.7577       10.5543       3.05455        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.71164       8.30877       11.3441       8.91233       2.30988        10     
[art]:TriggerResults:TriggerResultInserter        3.5342e-05    5.81919e-05   0.000121471   4.83515e-05   2.47946e-05      10     
end_path:out1:RootOutput                           7.461e-06    8.5349e-06    1.2718e-05     8.158e-06    1.44215e-06      10     
end_path:out1:RootOutput(write)                     2.66219       3.6315        4.21833       3.63039      0.503956        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5223.18 MB
  Peak resident set size usage (VmHWM): 4421.13 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.08 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10753372
-rw-r--r--. 1 dune009 dune 2500504547 Nov 24 03:38 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2372051905 Nov 24 03:35 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2207639988 Nov 24 05:00 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1967434895 Nov 24 00:35 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1883664164 Nov 24 00:17 g4_stage1.root
-rw-r--r--. 1 dune009 dune   79545750 Nov 24 04:55 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242972 Nov 24 05:00 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 24 05:00 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 24 05:00 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 24 05:00 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 24 03:38 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 24 00:35 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 24 05:00 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 24 00:10 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 24 05:00 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 24 03:44 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 24 03:38 debugprod.log
justIN time: 2026-02-08 19:59:59 UTC       justIN version: 01.06.00