justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271065.0@dunegpschedd02.fnal.gov

Jobsub ID271065.0@dunegpschedd02.fnal.gov
Workflow ID11793
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 06:13:01
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2026-01-13 11:26:40
From worker nodeHostnamewn-lb-02.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 06:13:59
Input filesmonte-carlo-011793-000001
JobscriptExit code0
Real time5h (18632s)
CPU time5h (18425s = 98%)
Max RSS bytes4403236864 (4199 MiB)
Outputting started2026-01-13 11:24:32
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11793/1/001/reco_stage1.root
Finished2026-01-13 11:26:40
Saved logsjustin-logs:271065.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 141960 traces, 0 summary
Retagger: tagging trace set: gauss with 110651 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 110651 traces tagged "gauss"
FrameSaver: q=3.9066e+06 n=2017239 tag=gauss
wclsFrameSaver: saving 141960 traces tagged "wiener"
FrameSaver: q=4.1288e+06 n=1811589 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33920 XUs and 31470 XVs -> 294 XUVs
C:0 T:1 57907 XUs and 51609 XVs -> 3161 XUVs
C:0 T:2 166673 XUs and 180428 XVs -> 13936 XUVs
C:0 T:3 69370 XUs and 60643 XVs -> 835 XUVs
C:0 T:4 6539 XUs and 10361 XVs -> 52 XUVs
C:0 T:5 24335 XUs and 27622 XVs -> 2635 XUVs
C:0 T:6 139509 XUs and 142832 XVs -> 7993 XUVs
C:0 T:7 51245 XUs and 53428 XVs -> 718 XUVs
C:0 T:8 5847 XUs and 9218 XVs -> 148 XUVs
C:0 T:9 50892 XUs and 90493 XVs -> 7242 XUVs
C:0 T:10 42487 XUs and 34798 XVs -> 4933 XUVs
C:0 T:11 293 XUs and 214 XVs -> 12 XUVs
41959 XUVs total
12561 collection wire objects
37766 potential space points
Neighbour search...
4704358 tests to find 2074568 neighbours
Iterating with no regularization...
Begin: 7.37005e+07
0 6.15738e+07
1 6.06282e+07
2 6.04899e+07
3 6.04485e+07
Now with regularization...
Begin: 4.96846e+07
0 4.96481e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411710752
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411710752
fLifetime: 30000 [us]
13-Jan-2026 12:24:29 CET  Closed output file "reco_stage1.root"
13-Jan-2026 12:24:29 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          395.315       554.581       801.612       549.531       106.187        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00132514    0.00942583     0.0309011    0.00731263     0.0078571       10     
reco:ophit:OpHitFinder                             0.382416      0.461604      0.546003      0.465408      0.0479516       10     
reco:opflash:OpFlashFinder                         0.272537      0.350156       0.41741       0.34415      0.046625        10     
reco:opslicer:OpSlicer                             0.769758      0.940617       1.10933       0.93813      0.110411        10     
reco:caldata:DataPrepModule                         6.40736       6.58852       7.19315       6.5658       0.215124        10     
reco:wclsdatasp:WireCellToolkit                     45.4898       60.2165       69.7005       60.6739       8.37963        10     
reco:gaushit:GausHitFinder                          1.2353        1.56577       1.93559       1.5594       0.194809        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000543631   0.000832999   0.00156509    0.000768352   0.000290393      10     
reco:reco3d:SpacePointSolver                        14.1924       24.8524       42.2982       24.4595       7.10059        10     
reco:hitpdune:DisambigFromSpacePoints               1.48122       2.16903       3.12388       2.14387      0.445626        10     
reco:pandora:StandardPandora                        30.8529       59.3071       114.894       55.6899       21.6849        10     
reco:pandoraWriter:StandardPandora                 0.473067      0.614862      0.799599      0.634499      0.0844241       10     
reco:pandoraTrack:LArPandoraTrackCreation           7.21673       11.0039       15.2774       10.5242       2.69485        10     
reco:pandoraShower:LArPandoraShowerCreation         8.03108       13.739        22.9195       12.1263       4.6796         10     
reco:pandoracalo:Calorimetry                        3.35958       5.70467       10.2687       5.30127       1.93846        10     
reco:pandoracalonosce:Calorimetry                   3.06163       5.32183       9.28547       5.0702        1.74352        10     
reco:pandorapid:Chi2ParticleID                    0.00557852    0.00713032    0.00968945    0.00691301    0.00105254       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0481272     0.0611062     0.0744023     0.0618728    0.00768915       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00560772    0.00692275    0.00819095    0.00701016    0.000795196      10     
reco:pandoraShowercalo:ShowerCalorimetry            7.11939       12.2762       21.4699       11.2458       4.20403        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       6.66509       12.2187       23.0317       11.0474       4.63313        10     
reco:emtrkmichelid:EmTrackMichelId                  180.904       224.949       277.386       229.734       26.8027        10     
reco:crttag:SingleCRTMatchingProducer              0.0179493     0.0280292     0.0460294      0.02662     0.00792056       10     
reco:crtreco:TwoCRTMatchingProducer                0.013768      0.0227189     0.0334982     0.0208169     0.0068391       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.75042       1.12887       1.48738       1.14814      0.203115        10     
reco:pandora2Track:LArPandoraTrackCreation          22.0037       36.0854       60.7366       34.5178       10.7385        10     
reco:pandora2calo:Calorimetry                       11.3337       17.4929       26.4481       17.626        4.2485         10     
reco:pandora2calonosce:Calorimetry                  8.9923        15.9819       24.5382       16.0182       4.36507        10     
reco:pandora2pid:Chi2ParticleID                    0.0127022     0.0164063     0.0196828     0.0164922    0.00178076       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0625818     0.0792616     0.0955159     0.0806895    0.00948274       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0127577     0.0162487     0.019911      0.0167159    0.00206813       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.99746       12.1993       19.9975       11.8733       3.56057        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.11378       13.6395       24.2746       13.6425       4.35492        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.04318       11.2864       17.8426       10.8189       3.11006        10     
[art]:TriggerResults:TriggerResultInserter        4.4496e-05    6.1952e-05    0.000128592   4.8793e-05    2.59115e-05      10     
end_path:out1:RootOutput                           8.061e-06    1.29595e-05   3.5644e-05     8.939e-06    8.25739e-06      10     
end_path:out1:RootOutput(write)                     3.12438       3.89901       4.91025       3.96653      0.532399        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5211.79 MB
  Peak resident set size usage (VmHWM): 4403.24 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.11 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11199216
-rw-r--r--. 1 dune009 dune 2596298665 Jan 13 10:49 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2460963450 Jan 13 10:46 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2316270483 Jan 13 12:24 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2043407568 Jan 13 07:39 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1966806443 Jan 13 07:22 g4_stage1.root
-rw-r--r--. 1 dune009 dune   83636760 Jan 13 12:17 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243468 Jan 13 12:24 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 13 12:24 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 13 12:24 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 13 12:24 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 13 10:49 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 07:39 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 12:24 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 13 07:14 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 13 12:24 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 13 10:57 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 13 10:49 debugprod.log
justIN time: 2026-02-06 07:39:15 UTC       justIN version: 01.06.00