justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 293577.1@dunegpschedd02.fnal.gov

Jobsub ID293577.1@dunegpschedd02.fnal.gov
Workflow ID12249
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-22 14:31:06
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2026-01-22 19:43:47
From worker nodeHostnamewn-lb-03.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-22 14:32:24
Input filesmonte-carlo-012249-000001
JobscriptExit code0
Real time5h (18576s)
CPU time5h (18371s = 98%)
Max RSS bytes4343967744 (4142 MiB)
Outputting started2026-01-22 19:42:02
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12249/1/001/reco_stage1.root
Finished2026-01-22 19:43:47
Saved logsjustin-logs:293577.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 144423 traces, 0 summary
Retagger: tagging trace set: gauss with 111161 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 111161 traces tagged "gauss"
FrameSaver: q=4.0515e+06 n=2047359 tag=gauss
wclsFrameSaver: saving 144423 traces tagged "wiener"
FrameSaver: q=4.27721e+06 n=1832875 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 54807 XUs and 37275 XVs -> 684 XUVs
C:0 T:1 166728 XUs and 77579 XVs -> 16551 XUVs
C:0 T:2 131218 XUs and 129995 XVs -> 4974 XUVs
C:0 T:3 65020 XUs and 53502 XVs -> 532 XUVs
C:0 T:4 12337 XUs and 9660 XVs -> 142 XUVs
C:0 T:5 78746 XUs and 40733 XVs -> 8868 XUVs
C:0 T:6 157574 XUs and 173534 XVs -> 13362 XUVs
C:0 T:7 61268 XUs and 62063 XVs -> 948 XUVs
C:0 T:8 7928 XUs and 8295 XVs -> 98 XUVs
C:0 T:9 70803 XUs and 195011 XVs -> 10733 XUVs
C:0 T:10 40951 XUs and 35445 XVs -> 9206 XUVs
C:0 T:11 362 XUs and 322 XVs -> 47 XUVs
66145 XUVs total
12692 collection wire objects
59633 potential space points
Neighbour search...
10705999 tests to find 4786268 neighbours
Iterating with no regularization...
Begin: 9.75266e+07
0 8.22763e+07
1 8.13723e+07
2 8.12481e+07
3 8.12138e+07
Now with regularization...
Begin: 6.9911e+07
0 6.98904e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411898811
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411898811
fLifetime: 30000 [us]
22-Jan-2026 20:41:58 CET  Closed output file "reco_stage1.root"
22-Jan-2026 20:41:58 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          352.442       641.829       859.038       639.064       139.785        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00159412    0.00817712     0.0124787    0.00824685    0.00342499       10     
reco:ophit:OpHitFinder                             0.369937      0.417717      0.473071      0.414652      0.0294594       10     
reco:opflash:OpFlashFinder                         0.201448       0.29903      0.392457      0.306195      0.0542121       10     
reco:opslicer:OpSlicer                             0.699056      0.829704       1.02162      0.790271      0.104492        10     
reco:caldata:DataPrepModule                         6.44686       6.72807       7.25982       6.67423      0.243552        10     
reco:wclsdatasp:WireCellToolkit                     49.3254       67.2701       75.2957       69.5103       8.79156        10     
reco:gaushit:GausHitFinder                          1.01784        1.533        2.12025       1.5075       0.299159        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000929295   0.00137237    0.00207261    0.00135775    0.000300177      10     
reco:reco3d:SpacePointSolver                        10.5627       27.1413       39.9741       26.0427       9.70202        10     
reco:hitpdune:DisambigFromSpacePoints               1.03837       1.95728       2.77317       1.83772      0.551934        10     
reco:pandora:StandardPandora                        30.3832       45.9201       67.8507       43.0947       12.2086        10     
reco:pandoraWriter:StandardPandora                 0.578683      0.735676      0.935579      0.719701      0.0972577       10     
reco:pandoraTrack:LArPandoraTrackCreation           8.92322       16.2529       23.4967       15.9251       4.53602        10     
reco:pandoraShower:LArPandoraShowerCreation         8.14917       20.6644       30.1586       19.9262       6.40252        10     
reco:pandoracalo:Calorimetry                        3.59237        7.964        11.2926       8.09533       2.17313        10     
reco:pandoracalonosce:Calorimetry                   2.79209       7.40289       11.7508       6.75065       2.45035        10     
reco:pandorapid:Chi2ParticleID                    0.00603106    0.00732842     0.0092997    0.00693969    0.000987523      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0443728     0.0635812     0.0759838     0.0633192    0.00908591       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00643491    0.00714659    0.00799814    0.00713082    0.000556186      10     
reco:pandoraShowercalo:ShowerCalorimetry            5.43151       18.8129       32.9064       18.2015       6.64598        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.90228       18.1607       32.0851       17.973        6.63451        10     
reco:emtrkmichelid:EmTrackMichelId                  160.945       224.337       275.269       218.882        32.25         10     
reco:crttag:SingleCRTMatchingProducer              0.0311919     0.0489203     0.0634314     0.0505668     0.0100377       10     
reco:crtreco:TwoCRTMatchingProducer                 0.02508      0.0396147     0.0543521     0.0414668     0.0105438       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.802789       1.15274       1.51172       1.12535      0.205655        10     
reco:pandora2Track:LArPandoraTrackCreation          17.7784       53.4521       90.9513       48.5547       19.3738        10     
reco:pandora2calo:Calorimetry                       6.80308       26.8267       42.4681       25.0024       9.47429        10     
reco:pandora2calonosce:Calorimetry                  6.0176        26.3128       43.1211       23.7103       11.207         10     
reco:pandora2pid:Chi2ParticleID                   0.00944885     0.015946      0.0211358     0.016101      0.003093        10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0530634      0.0817       0.110684      0.0815664     0.0144906       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00919277     0.0160161     0.0209858     0.0155118    0.00342339       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.63165       21.8437       38.4309       19.7219       8.21578        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.67253       22.4497       41.2948       20.9988       9.46475        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.47592       18.9651       38.0798       17.6352       8.46193        10     
[art]:TriggerResults:TriggerResultInserter        5.8348e-05    9.29514e-05   0.000163846   7.2008e-05    3.59364e-05      10     
end_path:out1:RootOutput                           9.835e-06    1.26148e-05   1.8318e-05    1.09165e-05   2.92577e-06      10     
end_path:out1:RootOutput(write)                     2.45175       3.62997       4.58673       3.56035      0.576948        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5076.98 MB
  Peak resident set size usage (VmHWM): 4271.36 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.06 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10144312
-rw-r--r--. 1 dune009 dune 2370547000 Jan 22 18:52 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2247551400 Jan 22 18:50 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2060456752 Jan 22 20:41 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1850128563 Jan 22 15:58 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1782922986 Jan 22 15:40 g4_stage1.root
-rw-r--r--. 1 dune009 dune   75554070 Jan 22 20:30 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242703 Jan 22 20:42 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 22 20:41 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 22 20:41 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 22 20:41 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 22 18:52 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 22 15:58 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 22 20:41 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 22 15:32 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 22 20:42 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 22 18:58 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 22 18:52 debugprod.log
justIN time: 2026-02-05 08:27:49 UTC       justIN version: 01.06.00