justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 278308.1@dunegpschedd01.fnal.gov

Jobsub ID278308.1@dunegpschedd01.fnal.gov
Workflow ID11928
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-14 22:13:04
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-15 02:12:38
From worker nodeHostnamewn-da-12.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-14 22:13:59
Input filesmonte-carlo-011928-000001
JobscriptExit code0
Real time3h (14235s)
CPU time3h (14122s = 99%)
Max RSS bytes4393172992 (4189 MiB)
Outputting started2026-01-15 02:11:15
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11928/1/001/reco_stage1.root
Finished2026-01-15 02:12:38
Saved logsjustin-logs:278308.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 184168 traces, 0 summary
Retagger: tagging trace set: gauss with 136698 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 136698 traces tagged "gauss"
FrameSaver: q=4.71478e+06 n=2420644 tag=gauss
wclsFrameSaver: saving 184168 traces tagged "wiener"
FrameSaver: q=4.99766e+06 n=2134676 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 39309 XUs and 36489 XVs -> 919 XUVs
C:0 T:1 124518 XUs and 137460 XVs -> 28726 XUVs
C:0 T:2 88475 XUs and 107736 XVs -> 4757 XUVs
C:0 T:3 40967 XUs and 41009 XVs -> 237 XUVs
C:0 T:4 17239 XUs and 18799 XVs -> 361 XUVs
C:0 T:5 153585 XUs and 183445 XVs -> 30713 XUVs
C:0 T:6 204848 XUs and 301349 XVs -> 13006 XUVs
C:0 T:7 69624 XUs and 91030 XVs -> 1249 XUVs
C:0 T:8 7488 XUs and 8985 XVs -> 198 XUVs
C:0 T:9 57973 XUs and 92738 XVs -> 11421 XUVs
C:0 T:10 66301 XUs and 82730 XVs -> 8348 XUVs
C:0 T:11 230 XUs and 266 XVs -> 22 XUVs
99957 XUVs total
17316 collection wire objects
92013 potential space points
Neighbour search...
23432719 tests to find 9380928 neighbours
Iterating with no regularization...
Begin: 9.77999e+07
0 8.03895e+07
1 7.91288e+07
2 7.8947e+07
3 7.88913e+07
Now with regularization...
Begin: 6.59584e+07
0 6.586e+07
1 6.5802e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411744339
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411744339
fLifetime: 30000 [us]
15-Jan-2026 03:11:13 CET  Closed output file "reco_stage1.root"
15-Jan-2026 03:11:13 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          305.27        431.631       589.09        424.087       87.5051        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000921848   0.00479692    0.00844389    0.00522124    0.00244343       10     
reco:ophit:OpHitFinder                             0.238979      0.302438      0.357809       0.32555      0.0443985       10     
reco:opflash:OpFlashFinder                         0.141527       0.22803       0.28844      0.250535      0.0561185       10     
reco:opslicer:OpSlicer                             0.485639      0.717261      0.895541      0.778514      0.158806        10     
reco:caldata:DataPrepModule                         4.3403        4.76424       5.27257       4.78092       0.2897         10     
reco:wclsdatasp:WireCellToolkit                     36.7199       55.1721       67.2513       57.0843       10.2762        10     
reco:gaushit:GausHitFinder                         0.979478       1.34703       1.73346       1.38013      0.236404        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000438216   0.000665993   0.000826656   0.000683052   0.00012335       10     
reco:reco3d:SpacePointSolver                        7.49742       16.0922       30.4486       14.9353       6.07704        10     
reco:hitpdune:DisambigFromSpacePoints              0.940517       1.78021       2.58773       1.83994      0.528178        10     
reco:pandora:StandardPandora                        24.066        43.3243       70.2395       40.9802       14.0743        10     
reco:pandoraWriter:StandardPandora                 0.326671      0.451116      0.581105      0.459077      0.081332        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.55726       5.36546       7.40364       5.43712       1.41791        10     
reco:pandoraShower:LArPandoraShowerCreation         2.90957       6.26062       10.0473       5.96714       2.33504        10     
reco:pandoracalo:Calorimetry                        1.39717       2.96382       4.27173       2.94703       0.83437        10     
reco:pandoracalonosce:Calorimetry                   1.27506       2.75219       3.96112       2.73311      0.791658        10     
reco:pandorapid:Chi2ParticleID                    0.00447003    0.00691927     0.0143606    0.00639872    0.00259054       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0390356      0.05341      0.0689516     0.0529394    0.00947817       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0046035    0.00602831    0.00693725    0.00628826    0.000692671      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.74801       6.29057       9.6267        6.09691       2.37328        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.58414       6.04972       9.60385       5.74314       2.38196        10     
reco:emtrkmichelid:EmTrackMichelId                  159.57        222.212       281.482       226.227       39.8955        10     
reco:crttag:SingleCRTMatchingProducer             0.00857068     0.0161542     0.025152      0.015524     0.00447655       10     
reco:crtreco:TwoCRTMatchingProducer               0.00678019     0.0105775     0.0188507     0.0101219     0.0033623       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.623529      0.916888       1.19372      0.935932      0.185611        10     
reco:pandora2Track:LArPandoraTrackCreation          7.35622       17.1097       28.0072       16.6491       6.1597         10     
reco:pandora2calo:Calorimetry                       3.44541       8.00587       12.7192       7.57895       2.71634        10     
reco:pandora2calonosce:Calorimetry                  3.35014       7.54759       12.3278       7.16848       2.6157         10     
reco:pandora2pid:Chi2ParticleID                    0.0109212     0.0152046     0.0176523     0.0160399     0.0021877       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0487877     0.0673353     0.0836428     0.0663406     0.0113422       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0107599     0.0150397     0.0177169     0.0158343    0.00228934       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.99943       6.01937       9.90473       5.73663       2.00112        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.21545       6.65222       11.1447       6.30717       2.23036        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.94904       5.78029       9.49713       5.14521       1.87754        10     
[art]:TriggerResults:TriggerResultInserter        2.6451e-05    3.75918e-05   9.9159e-05    3.0443e-05    2.06992e-05      10     
end_path:out1:RootOutput                            5.1e-06     9.2066e-06    3.0549e-05    6.5675e-06    7.28863e-06      10     
end_path:out1:RootOutput(write)                     2.22733       3.17705       4.0476        3.30432       0.57826        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5198 MB
  Peak resident set size usage (VmHWM): 4393.17 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.97 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11100088
-rw-r--r--. 1 dune009 dune 2571382208 Jan 15 01:57 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2435279811 Jan 15 01:55 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2302464870 Jan 15 03:11 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2031636150 Jan 14 23:35 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1939854336 Jan 14 23:20 g4_stage1.root
-rw-r--r--. 1 dune009 dune   85259590 Jan 15 03:04 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242827 Jan 15 03:11 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 15 03:11 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 15 03:11 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 15 03:11 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 15 01:57 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 14 23:35 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 15 03:11 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 14 23:14 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 15 03:11 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 15 02:03 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 15 01:57 debugprod.log
justIN time: 2026-02-06 02:07:26 UTC       justIN version: 01.06.00