justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263551.1@dunegpschedd02.fnal.gov

Jobsub ID263551.1@dunegpschedd02.fnal.gov
Workflow ID11293
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-20 06:19:48
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-12-20 10:18:15
From worker nodeHostnamewn-da-02.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-20 06:21:17
Input filesmonte-carlo-011293-000001
JobscriptExit code0
Real time3h (14129s)
CPU time3h (14045s = 99%)
Max RSS bytes4457951232 (4251 MiB)
Outputting started2025-12-20 10:16:47
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11293/1/001/reco_stage1.root
Finished2025-12-20 10:18:15
Saved logsjustin-logs:263551.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Service:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 120914 traces, 0 summary
Retagger: tagging trace set: gauss with 88554 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88554 traces tagged "gauss"
FrameSaver: q=2.96452e+06 n=1537033 tag=gauss
wclsFrameSaver: saving 120914 traces tagged "wiener"
FrameSaver: q=3.1485e+06 n=1332481 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 16975 XUs and 19279 XVs -> 130 XUVs
C:0 T:1 35379 XUs and 37019 XVs -> 3551 XUVs
C:0 T:2 91606 XUs and 119960 XVs -> 7287 XUVs
C:0 T:3 44932 XUs and 48843 XVs -> 492 XUVs
C:0 T:4 8400 XUs and 10038 XVs -> 152 XUVs
C:0 T:5 91193 XUs and 122403 XVs -> 19106 XUVs
C:0 T:6 109584 XUs and 224305 XVs -> 18874 XUVs
C:0 T:7 27585 XUs and 39848 XVs -> 421 XUVs
C:0 T:8 7436 XUs and 8035 XVs -> 141 XUVs
C:0 T:9 154196 XUs and 85573 XVs -> 11291 XUVs
C:0 T:10 111759 XUs and 93685 XVs -> 22431 XUVs
C:0 T:11 102 XUs and 82 XVs -> 14 XUVs
83890 XUVs total
12623 collection wire objects
77442 potential space points
Neighbour search...
20362004 tests to find 8455728 neighbours
Iterating with no regularization...
Begin: 9.54456e+07
0 6.20391e+07
1 6.06468e+07
2 6.0461e+07
3 6.03905e+07
4 6.03518e+07
Now with regularization...
Begin: 5.16781e+07
0 5.16059e+07
1 5.15482e+07
2 5.1501e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411225992
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411225992
fLifetime: 30000 [us]
20-Dec-2025 11:16:45 CET  Closed output file "reco_stage1.root"
20-Dec-2025 11:16:45 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          320.584       476.687       655.121       461.494       95.9063        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00114007    0.00546593    0.00997515     0.0052255    0.00294941       10     
reco:ophit:OpHitFinder                              0.23811      0.305554      0.362785       0.30989      0.0386346       10     
reco:opflash:OpFlashFinder                         0.145106      0.219233      0.302898       0.2261       0.0463146       10     
reco:opslicer:OpSlicer                             0.442199      0.693744      0.856725      0.705298      0.126013        10     
reco:caldata:DataPrepModule                         4.89716       5.09213       5.39709       5.09182      0.133675        10     
reco:wclsdatasp:WireCellToolkit                     37.4893       55.5955       71.1714       60.0491       13.5865        10     
reco:gaushit:GausHitFinder                          1.0624        1.41769       1.72828       1.43607      0.204365        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00055045    0.000899776   0.00137788    0.000836291   0.000256998      10     
reco:reco3d:SpacePointSolver                        11.4432       18.8798       32.4212       17.3733       5.62042        10     
reco:hitpdune:DisambigFromSpacePoints               1.08458       1.77452       2.54929       1.74421      0.459299        10     
reco:pandora:StandardPandora                        24.6938       41.4064       64.4731       38.2028       11.9842        10     
reco:pandoraWriter:StandardPandora                 0.391783      0.496498      0.615314      0.512402      0.0763264       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.41156       7.13726       12.4754       7.07265       2.33378        10     
reco:pandoraShower:LArPandoraShowerCreation         3.45973       8.73928       15.7823       8.34265       3.57074        10     
reco:pandoracalo:Calorimetry                        2.3965        3.79678       6.64711       3.68186       1.20851        10     
reco:pandoracalonosce:Calorimetry                   1.92289       3.62162       6.16788       3.73298       1.1496         10     
reco:pandorapid:Chi2ParticleID                    0.00563118    0.00670426    0.00812501    0.00662701    0.000793543      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0447258     0.0587178     0.0737982     0.0583175    0.00910575       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00535805    0.00635512    0.00813218    0.00618779    0.000823624      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.93505       8.35351       14.3808       8.85957       3.12296        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.76325       8.18853       14.0566       8.19243       3.31615        10     
reco:emtrkmichelid:EmTrackMichelId                  178.093         234         290.137       235.213       35.0847        10     
reco:crttag:SingleCRTMatchingProducer              0.0104924     0.0196755     0.0245757     0.0204036    0.00445883       10     
reco:crtreco:TwoCRTMatchingProducer               0.00840205     0.0147572     0.0218391     0.0146402    0.00450127       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.610056       0.93541       1.16698      0.952189      0.166006        10     
reco:pandora2Track:LArPandoraTrackCreation          13.4529       24.1466       42.6473       21.5619       8.82981        10     
reco:pandora2calo:Calorimetry                       5.69024       11.7446       18.5543       10.4929       4.26414        10     
reco:pandora2calonosce:Calorimetry                  5.27368       10.8537       19.0253       10.3839       3.90322        10     
reco:pandora2pid:Chi2ParticleID                     0.01325      0.0164704     0.020978      0.016183     0.00243612       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0594209     0.0761492     0.0983337     0.0731859     0.0120965       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0133955     0.0161516     0.0205126     0.0157542    0.00209493       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.77671       8.41445       14.3662       7.39814       3.08022        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.56241       9.11898       14.564        8.06754       3.53411        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.82166       7.99333       14.3539       7.81456       2.8897         10     
[art]:TriggerResults:TriggerResultInserter        3.6139e-05    5.06628e-05   0.000120189   4.31475e-05   2.34328e-05      10     
end_path:out1:RootOutput                           7.424e-06    8.3687e-06    1.2564e-05    8.0155e-06    1.42778e-06      10     
end_path:out1:RootOutput(write)                     2.45742       3.33934       4.12299       3.39117       0.5062         10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5261.35 MB
  Peak resident set size usage (VmHWM): 4457.95 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.17 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11151460
-rw-r--r--. 1 dune009 dune 2594468249 Dec 20 09:55 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2464861926 Dec 20 09:52 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2316602737 Dec 20 11:16 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2022521105 Dec 20 07:41 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1936701535 Dec 20 07:27 g4_stage1.root
-rw-r--r--. 1 dune009 dune   83328320 Dec 20 11:12 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242860 Dec 20 11:16 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Dec 20 11:16 mem.db
-rw-r--r--. 1 dune009 dune      62376 Dec 20 11:16 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Dec 20 11:16 time.db
-rw-r--r--. 1 dune009 dune        519 Dec 20 09:55 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 20 07:41 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 20 11:16 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Dec 20 07:21 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Dec 20 11:16 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Dec 20 10:01 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Dec 20 09:55 debugprod.log
justIN time: 2026-02-07 06:27:41 UTC       justIN version: 01.06.00