justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279646.0@dunegpschedd01.fnal.gov

Jobsub ID279646.0@dunegpschedd01.fnal.gov
Workflow ID11991
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-16 21:23:36
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-17 01:50:14
From worker nodeHostnamewn-la-06.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-16 21:24:24
Input filesmonte-carlo-011991-000001
JobscriptExit code0
Real time4h (15839s)
CPU time4h (15699s = 99%)
Max RSS bytes4228079616 (4032 MiB)
Outputting started2026-01-17 01:48:24
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11991/1/001/reco_stage1.root
Finished2026-01-17 01:50:14
Saved logsjustin-logs:279646.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 123507 traces, 0 summary
Retagger: tagging trace set: gauss with 96875 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96875 traces tagged "gauss"
FrameSaver: q=3.35914e+06 n=1733083 tag=gauss
wclsFrameSaver: saving 123507 traces tagged "wiener"
FrameSaver: q=3.528e+06 n=1543306 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 26926 XUs and 18954 XVs -> 139 XUVs
C:0 T:1 48541 XUs and 36496 XVs -> 3884 XUVs
C:0 T:2 116858 XUs and 148587 XVs -> 6249 XUVs
C:0 T:3 56235 XUs and 59423 XVs -> 568 XUVs
C:0 T:4 8882 XUs and 8342 XVs -> 52 XUVs
C:0 T:5 26660 XUs and 17782 XVs -> 1993 XUVs
C:0 T:6 56276 XUs and 90899 XVs -> 2688 XUVs
C:0 T:8 10376 XUs and 11452 XVs -> 111 XUVs
C:0 T:9 58260 XUs and 76925 XVs -> 4673 XUVs
C:0 T:10 25856 XUs and 18196 XVs -> 2833 XUVs
C:0 T:11 80 XUs and 45 XVs -> 11 XUVs
23201 XUVs total
10404 collection wire objects
21518 potential space points
Neighbour search...
751268 tests to find 417832 neighbours
Iterating with no regularization...
Begin: 6.86732e+07
0 5.97366e+07
1 5.91515e+07
2 5.90801e+07
3 5.90608e+07
Now with regularization...
Begin: 4.98172e+07
0 4.97969e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411783820
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411783820
fLifetime: 30000 [us]
17-Jan-2026 02:48:21 CET  Closed output file "reco_stage1.root"
17-Jan-2026 02:48:21 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          354.994       490.057       686.616       502.197       107.012        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00135643    0.00603638    0.00882744    0.00711327    0.00260884       10     
reco:ophit:OpHitFinder                             0.381582      0.420936       0.48321      0.406627      0.034233        10     
reco:opflash:OpFlashFinder                         0.245513      0.289628      0.361747      0.281473      0.0383067       10     
reco:opslicer:OpSlicer                             0.717322      0.848155       1.02138       0.84105      0.107441        10     
reco:caldata:DataPrepModule                         6.22476       6.48505       7.15234       6.41427      0.265268        10     
reco:wclsdatasp:WireCellToolkit                     46.6148       63.4029       71.3828       66.4346       8.35331        10     
reco:gaushit:GausHitFinder                          1.02945       1.36676       1.75036       1.32985      0.243919        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000488832   0.000784195   0.00110813    0.000698816   0.000226712      10     
reco:reco3d:SpacePointSolver                        9.72568        21.6         33.7691       22.7168       7.42959        10     
reco:hitpdune:DisambigFromSpacePoints               1.09974       1.81329       2.94901       1.62264      0.575062        10     
reco:pandora:StandardPandora                        22.4024       41.3713       68.7157       38.6297       14.7358        10     
reco:pandoraWriter:StandardPandora                 0.394922      0.512811      0.686634      0.502128      0.0998437       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.77113       8.66305       13.9524       7.97863       3.61465        10     
reco:pandoraShower:LArPandoraShowerCreation         3.67572       10.4191       20.6929       9.05846       5.84293        10     
reco:pandoracalo:Calorimetry                        1.97574       4.48824       7.56362       4.06902       1.9435         10     
reco:pandoracalonosce:Calorimetry                   1.85695       4.3017        7.51683       3.75926       1.9655         10     
reco:pandorapid:Chi2ParticleID                    0.00541003    0.00678696    0.00874487    0.00673908    0.00103196       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP          0.04151      0.0562688     0.0792314     0.0552202     0.0114978       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00520641    0.00649299    0.00771756    0.00644912    0.000873989      10     
reco:pandoraShowercalo:ShowerCalorimetry            3.51407       9.47866       17.6165       8.60755       4.99968        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        3.44         9.41552       19.0201       9.0781        5.10717        10     
reco:emtrkmichelid:EmTrackMichelId                  172.614       219.951       281.313       215.018       32.7254        10     
reco:crttag:SingleCRTMatchingProducer              0.010076      0.0222031     0.0332199     0.0240462    0.00725694       10     
reco:crtreco:TwoCRTMatchingProducer               0.00852586     0.0184239     0.0309065     0.0186826    0.00763385       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.850543       1.09337       1.7026        1.02499      0.234582        10     
reco:pandora2Track:LArPandoraTrackCreation          11.2271       26.5039       48.8659       22.8428       13.1868        10     
reco:pandora2calo:Calorimetry                       5.0102        12.8021       23.5748       11.3913       6.61741        10     
reco:pandora2calonosce:Calorimetry                  4.73018       12.192        22.6205       10.8255       6.2233         10     
reco:pandora2pid:Chi2ParticleID                    0.0113723     0.0148696     0.018221      0.0153799    0.00234572       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0533979     0.070733      0.0980904     0.0718887     0.0133474       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0115927     0.0149035     0.0186129     0.0151117     0.0023159       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.95019       9.75133       17.9733       8.31706       5.01361        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.14451       10.4256       20.612        9.35193       5.5122         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.55318       8.54187       18.4313       7.65851       4.51099        10     
[art]:TriggerResults:TriggerResultInserter        4.3724e-05    5.89493e-05   0.000124026   5.09765e-05   2.28689e-05      10     
end_path:out1:RootOutput                           7.581e-06    9.0839e-06    1.4041e-05     8.528e-06    1.80699e-06      10     
end_path:out1:RootOutput(write)                     2.7216        3.43622       4.68293       3.32268       0.5948         10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5030.24 MB
  Peak resident set size usage (VmHWM): 4228.08 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.02 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10347304
-rw-r--r--. 1 dune009 dune 2423750574 Jan 17 01:24 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2297733150 Jan 17 01:22 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2109172381 Jan 17 02:48 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1889560696 Jan 16 22:48 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1800207449 Jan 16 22:31 g4_stage1.root
-rw-r--r--. 1 dune009 dune   74601580 Jan 17 02:42 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242732 Jan 17 02:48 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 17 02:48 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 17 02:48 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 17 02:48 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 17 01:24 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 16 22:48 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 17 02:48 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 16 22:24 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 17 02:48 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 17 01:32 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 17 01:24 debugprod.log
justIN time: 2026-02-05 23:28:10 UTC       justIN version: 01.06.00