justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237458.1@dunegpschedd02.fnal.gov

Jobsub ID237458.1@dunegpschedd02.fnal.gov
Workflow ID9597
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-04 19:30:26
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-04 23:58:51
From worker nodeHostnamewn-lb-09.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 19:31:17
Input filesmonte-carlo-009597-000001
JobscriptExit code0
Real time4h (15950s)
CPU time4h (15805s = 99%)
Max RSS bytes4554293248 (4343 MiB)
Outputting started2025-11-04 23:57:08
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09597/1/001/reco_stage1.root
Finished2025-11-04 23:58:51
Saved logsjustin-logs:237458.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

DigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 132460 traces, 0 summary
Retagger: tagging trace set: gauss with 97891 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 97891 traces tagged "gauss"
FrameSaver: q=3.21158e+06 n=1695785 tag=gauss
wclsFrameSaver: saving 132460 traces tagged "wiener"
FrameSaver: q=3.43564e+06 n=1489662 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21951 XUs and 22543 XVs -> 269 XUVs
C:0 T:1 48797 XUs and 69831 XVs -> 8090 XUVs
C:0 T:2 88007 XUs and 141092 XVs -> 5887 XUVs
C:0 T:3 41448 XUs and 54988 XVs -> 310 XUVs
C:0 T:4 6746 XUs and 9053 XVs -> 55 XUVs
C:0 T:5 44636 XUs and 73878 XVs -> 11193 XUVs
C:0 T:6 147619 XUs and 160859 XVs -> 9229 XUVs
C:0 T:7 53878 XUs and 51263 XVs -> 527 XUVs
C:0 T:8 4390 XUs and 6048 XVs -> 64 XUVs
C:0 T:9 27668 XUs and 49060 XVs -> 3083 XUVs
C:0 T:10 69267 XUs and 74346 XVs -> 14551 XUVs
C:0 T:11 237 XUs and 210 XVs -> 29 XUVs
53287 XUVs total
13533 collection wire objects
49490 potential space points
Neighbour search...
7882138 tests to find 3658974 neighbours
Iterating with no regularization...
Begin: 5.62909e+07
0 4.76711e+07
1 4.70403e+07
2 4.69428e+07
3 4.69097e+07
Now with regularization...
Begin: 3.76523e+07
0 3.75935e+07
1 3.75489e+07
2 3.75159e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410313837
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410313837
fLifetime: 30000 [us]
05-Nov-2025 00:57:06 CET  Closed output file "reco_stage1.root"
05-Nov-2025 00:57:06 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          438.559       579.445       759.227       579.854       99.5952        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00147591    0.00653868     0.0103287    0.00721504     0.0026991       10     
reco:ophit:OpHitFinder                             0.301013      0.413073      0.545897      0.438326      0.0772807       10     
reco:opflash:OpFlashFinder                          0.20524      0.300785      0.396828      0.314108      0.0671436       10     
reco:opslicer:OpSlicer                             0.529586      0.849604       1.09067      0.878724      0.195824        10     
reco:caldata:DataPrepModule                         4.87593       6.16529       7.26767       6.52195       0.76863        10     
reco:wclsdatasp:WireCellToolkit                     45.4406       62.7155       75.6275       62.246        11.0065        10     
reco:gaushit:GausHitFinder                          1.07252       1.55543       2.08263       1.45273      0.303247        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000856022    0.0010634     0.0012405    0.00109311    0.000134218      10     
reco:reco3d:SpacePointSolver                        13.2814       25.1461       48.9059       22.4689       10.2273        10     
reco:hitpdune:DisambigFromSpacePoints               1.43529       2.24343       3.36913       1.9679       0.688051        10     
reco:pandora:StandardPandora                        33.1827       58.3789       85.3513       56.1854       15.5901        10     
reco:pandoraWriter:StandardPandora                 0.540193      0.656865      0.970557      0.605039      0.134305        10     
reco:pandoraTrack:LArPandoraTrackCreation           8.94659       12.5082       17.5754       11.9778       2.94447        10     
reco:pandoraShower:LArPandoraShowerCreation         9.77911       16.4032       31.6904       15.3777       6.31134        10     
reco:pandoracalo:Calorimetry                        4.19756       5.99004       8.65567       5.62344       1.46209        10     
reco:pandoracalonosce:Calorimetry                   3.37313       5.69976       8.28635       5.50247       1.47599        10     
reco:pandorapid:Chi2ParticleID                    0.00517211    0.00700882     0.0086169    0.00691087    0.00099176       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0445504      0.06086      0.075603      0.0585864    0.00938257       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0052726    0.00682556    0.00855591     0.0068388    0.000952963      10     
reco:pandoraShowercalo:ShowerCalorimetry            9.2364        14.5348       21.4259       14.9226       4.30463        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       9.27107       13.6456       20.5426       12.833        3.8989         10     
reco:emtrkmichelid:EmTrackMichelId                  177.081       230.322       293.212       225.645       30.7802        10     
reco:crttag:SingleCRTMatchingProducer              0.0153974     0.0299406     0.0501328     0.0301177    0.00986529       10     
reco:crtreco:TwoCRTMatchingProducer                0.015124      0.0269161     0.0437821     0.024665     0.00852205       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.755665       1.08799       1.35135       1.05028      0.161494        10     
reco:pandora2Track:LArPandoraTrackCreation          25.3246       41.2326       59.8001       45.3391       10.9276        10     
reco:pandora2calo:Calorimetry                       11.1864       18.9515       26.8719       19.8429       5.10868        10     
reco:pandora2calonosce:Calorimetry                  10.1223       17.5467       24.3629       18.3088       4.88003        10     
reco:pandora2pid:Chi2ParticleID                    0.0123105     0.0163034     0.0203865     0.0155372    0.00249193       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0632418     0.0789576     0.0968824     0.0743977     0.0118529       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0129355     0.016316      0.0207066     0.0158968    0.00253879       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.65774       13.4893       19.9294       13.6047       4.48802        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.48431       13.7952       20.6685       12.5909       4.53696        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.68752       11.366        17.0618       10.6993       3.48663        10     
[art]:TriggerResults:TriggerResultInserter        4.3966e-05    6.4022e-05    0.00014633     5.019e-05    3.2921e-05       10     
end_path:out1:RootOutput                           7.441e-06     8.671e-06    1.3721e-05     8.292e-06    1.74745e-06      10     
end_path:out1:RootOutput(write)                     2.84802       3.8509        4.91975       3.55383      0.689511        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.64 MB
  Peak resident set size usage (VmHWM): 4554.29 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.81 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11334560
-rw-r--r--. 1 dune009 dune 2622708304 Nov  4 23:18 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2490595100 Nov  4 23:15 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2359659954 Nov  5 00:57 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2065927618 Nov  4 20:52 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1979174241 Nov  4 20:38 g4_stage1.root
-rw-r--r--. 1 dune009 dune   87914220 Nov  5 00:50 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243437 Nov  5 00:57 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  5 00:57 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  5 00:57 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  5 00:57 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  4 23:18 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  4 20:52 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  5 00:57 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  4 20:31 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  5 00:57 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  4 23:28 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  4 23:18 debugprod.log
justIN time: 2026-02-11 07:40:35 UTC       justIN version: 01.06.00