justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271486.1@dunegpschedd02.fnal.gov

Jobsub ID271486.1@dunegpschedd02.fnal.gov
Workflow ID11834
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 15:41:30
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-13 19:40:20
From worker nodeHostnamewn-db-07.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 15:42:22
Input filesmonte-carlo-011834-000001
JobscriptExit code0
Real time3h (14175s)
CPU time3h (14015s = 98%)
Max RSS bytes4383809536 (4180 MiB)
Outputting started2026-01-13 19:38:38
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11834/1/001/reco_stage1.root
Finished2026-01-13 19:40:20
Saved logsjustin-logs:271486.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 158221 traces, 0 summary
Retagger: tagging trace set: gauss with 120362 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 120362 traces tagged "gauss"
FrameSaver: q=4.19728e+06 n=2093861 tag=gauss
wclsFrameSaver: saving 158221 traces tagged "wiener"
FrameSaver: q=4.45029e+06 n=1856886 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 32781 XUs and 34458 XVs -> 246 XUVs
C:0 T:1 55828 XUs and 59096 XVs -> 4163 XUVs
C:0 T:2 73357 XUs and 109877 XVs -> 4058 XUVs
C:0 T:3 37848 XUs and 43229 XVs -> 266 XUVs
C:0 T:4 8568 XUs and 7843 XVs -> 45 XUVs
C:0 T:5 24583 XUs and 18717 XVs -> 1654 XUVs
C:0 T:6 192707 XUs and 198256 XVs -> 27381 XUVs
C:0 T:7 49955 XUs and 64669 XVs -> 1512 XUVs
C:0 T:8 8367 XUs and 7001 XVs -> 50 XUVs
C:0 T:9 43890 XUs and 49328 XVs -> 2830 XUVs
C:0 T:10 115835 XUs and 112559 XVs -> 17469 XUVs
C:0 T:11 732 XUs and 657 XVs -> 69 XUVs
59743 XUVs total
13074 collection wire objects
54260 potential space points
Neighbour search...
9827430 tests to find 4172646 neighbours
Iterating with no regularization...
Begin: 7.0134e+07
0 5.73901e+07
1 5.66156e+07
2 5.65056e+07
3 5.64744e+07
Now with regularization...
Begin: 4.56787e+07
0 4.56207e+07
1 4.55852e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411718688
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411718688
fLifetime: 30000 [us]
13-Jan-2026 20:38:35 CET  Closed output file "reco_stage1.root"
13-Jan-2026 20:38:35 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          343.499       465.492       560.47        489.897       63.1487        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000880778   0.00522678    0.00884387    0.00498939    0.00272518       10     
reco:ophit:OpHitFinder                             0.270869      0.316021      0.388969      0.316452      0.0364761       10     
reco:opflash:OpFlashFinder                         0.170652       0.23013      0.317583      0.223259      0.0430582       10     
reco:opslicer:OpSlicer                             0.584103      0.760649      0.914274      0.780487      0.110025        10     
reco:caldata:DataPrepModule                         4.87098       5.0011        5.47269       4.95552      0.173788        10     
reco:wclsdatasp:WireCellToolkit                     38.1458       70.3165       80.1787       77.4114       15.5925        10     
reco:gaushit:GausHitFinder                          1.00135       1.33867       1.61588       1.37272      0.176031        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000554818   0.000800625   0.00100489    0.000824877   0.000117609      10     
reco:reco3d:SpacePointSolver                        11.0646       15.986        21.2848       15.5301       2.89008        10     
reco:hitpdune:DisambigFromSpacePoints               1.11192       1.87684       2.72835       1.98916      0.488624        10     
reco:pandora:StandardPandora                        22.5796       43.3963       77.8562       41.4935       14.1432        10     
reco:pandoraWriter:StandardPandora                 0.328349      0.465825      0.556878      0.484862      0.0706115       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.30987       5.67853       8.00362       5.63993       1.48217        10     
reco:pandoraShower:LArPandoraShowerCreation         2.78423       6.71704       9.92963       6.97764       2.06739        10     
reco:pandoracalo:Calorimetry                        1.64447       3.05608       4.35163       3.05742      0.766142        10     
reco:pandoracalonosce:Calorimetry                   1.50413       2.8726        4.06909       2.85121      0.730938        10     
reco:pandorapid:Chi2ParticleID                    0.00494179    0.00665164    0.00931586     0.0063026    0.00123701       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.039446      0.0542062     0.064311      0.0561642    0.00710279       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00479533     0.0062312    0.00762301    0.00626313    0.000813745      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.75872       6.7047        10.3874       6.95415       2.03611        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.40886       6.49561       9.97745       6.72353       1.96295        10     
reco:emtrkmichelid:EmTrackMichelId                  176.288       233.814       283.478       244.903       32.6401        10     
reco:crttag:SingleCRTMatchingProducer              0.012307      0.0182269     0.0285748     0.0172679     0.0051847       10     
reco:crtreco:TwoCRTMatchingProducer               0.00729779     0.0116008     0.0147553     0.011644     0.00220476       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.654521      0.932449       1.19546      0.974735      0.154036        10     
reco:pandora2Track:LArPandoraTrackCreation          8.44947       18.6827       28.6087       18.7935       5.80849        10     
reco:pandora2calo:Calorimetry                       3.77216       8.90876       13.284        9.0026        2.60885        10     
reco:pandora2calonosce:Calorimetry                  3.55158       8.33261       11.9433       8.51319       2.36586        10     
reco:pandora2pid:Chi2ParticleID                    0.0119604     0.0166051     0.0230895     0.0166789    0.00313968       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0498368     0.070162      0.088266      0.0715374     0.0102536       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.011674      0.0163257     0.0213087     0.0166685    0.00281439       10     
reco:pandora2Shower:LArPandoraShowerCreation         3.271        6.58219       9.20677       6.58855       1.85776        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.21768       7.17897       9.85872       7.44024       1.8727         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.85531        6.219        8.64816       6.39726       1.48946        10     
[art]:TriggerResults:TriggerResultInserter        3.1089e-05    4.14973e-05   0.000104078   3.3464e-05    2.12168e-05      10     
end_path:out1:RootOutput                           6.312e-06    9.95529e-05   0.000556831   7.4245e-06    0.000188571      10     
end_path:out1:RootOutput(write)                     2.43946       3.26745       3.98212       3.42013      0.491517        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5394.78 MB
  Peak resident set size usage (VmHWM): 4383.81 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.12 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11319716
-rw-r--r--. 1 dune009 dune 2613790413 Jan 13 19:18 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2477929010 Jan 13 19:16 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2341206095 Jan 13 20:38 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2071814721 Jan 13 17:02 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2000800469 Jan 13 16:49 g4_stage1.root
-rw-r--r--. 1 dune009 dune   85232310 Jan 13 20:32 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242810 Jan 13 20:38 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 13 20:38 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 13 20:38 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 13 20:38 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 13 19:18 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 17:02 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 20:38 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 13 16:42 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 13 20:38 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 13 19:26 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 13 19:18 debugprod.log
justIN time: 2026-02-06 05:56:59 UTC       justIN version: 01.06.00