justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234101.0@dunegpschedd02.fnal.gov

Jobsub ID234101.0@dunegpschedd02.fnal.gov
Workflow ID9425
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-28 23:31:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-10-29 03:22:18
From worker nodeHostnamewn-lb-01.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 23:32:20
Input filesmonte-carlo-009425-000001
JobscriptExit code0
Real time3h (13688s)
CPU time3h (13512s = 98%)
Max RSS bytes4432199680 (4226 MiB)
Outputting started2025-10-29 03:20:29
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09425/1/001/reco_stage1.root
Finished2025-10-29 03:22:18
Saved logsjustin-logs:234101.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

hannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 178117 traces, 0 summary
Retagger: tagging trace set: gauss with 134045 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 134045 traces tagged "gauss"
FrameSaver: q=4.83622e+06 n=2406372 tag=gauss
wclsFrameSaver: saving 178117 traces tagged "wiener"
FrameSaver: q=5.12303e+06 n=2135602 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 43626 XUs and 47373 XVs -> 481 XUVs
C:0 T:1 97824 XUs and 88132 XVs -> 6234 XUVs
C:0 T:2 136217 XUs and 188491 XVs -> 9772 XUVs
C:0 T:3 63886 XUs and 64728 XVs -> 663 XUVs
C:0 T:4 16903 XUs and 17453 XVs -> 143 XUVs
C:0 T:5 63854 XUs and 60022 XVs -> 4339 XUVs
C:0 T:6 217393 XUs and 242441 XVs -> 16879 XUVs
C:0 T:7 74953 XUs and 73465 XVs -> 1499 XUVs
C:0 T:8 3969 XUs and 6231 XVs -> 47 XUVs
C:0 T:9 26625 XUs and 61019 XVs -> 4095 XUVs
C:0 T:10 102086 XUs and 67386 XVs -> 9331 XUVs
C:0 T:11 218 XUs and 135 XVs -> 28 XUVs
53511 XUVs total
16451 collection wire objects
47420 potential space points
Neighbour search...
4085336 tests to find 2023832 neighbours
Iterating with no regularization...
Begin: 9.40125e+07
0 7.42271e+07
1 7.28001e+07
2 7.25907e+07
3 7.25234e+07
Now with regularization...
Begin: 5.9096e+07
0 5.90397e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410176322
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410176322
fLifetime: 30000 [us]
29-Oct-2025 04:20:27 CET  Closed output file "reco_stage1.root"
29-Oct-2025 04:20:27 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          267.671       394.61        578.822       368.901       95.0473        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000987844   0.00569907     0.0100308    0.00588455    0.00291991       10     
reco:ophit:OpHitFinder                             0.293623      0.339016      0.396622      0.344196      0.030864        10     
reco:opflash:OpFlashFinder                         0.181506      0.250953      0.309992      0.259243      0.0396472       10     
reco:opslicer:OpSlicer                             0.557738      0.677886      0.823161      0.668565      0.0914479       10     
reco:caldata:DataPrepModule                         4.76386       4.97515       5.31411       4.93988      0.161277        10     
reco:wclsdatasp:WireCellToolkit                     34.8167       45.7709       55.7216       45.7253       6.9722         10     
reco:gaushit:GausHitFinder                         0.900306       1.17613       1.57284       1.12381      0.216853        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000433972   0.000780175   0.00102802    0.000854267   0.00019203       10     
reco:reco3d:SpacePointSolver                        9.91239       18.7192       28.5701       18.8707       5.4907         10     
reco:hitpdune:DisambigFromSpacePoints              0.939623       1.61927       2.63097        1.465       0.526628        10     
reco:pandora:StandardPandora                        18.4205       36.1388       65.8474       32.5802       14.8802        10     
reco:pandoraWriter:StandardPandora                 0.353405      0.457206      0.576434      0.446289      0.0706353       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.90216       6.79439       11.034        6.15732       2.2462         10     
reco:pandoraShower:LArPandoraShowerCreation         3.98911       8.71524       17.1057       7.78508       4.15012        10     
reco:pandoracalo:Calorimetry                        1.84357       3.53518       5.88487       3.39676       1.16626        10     
reco:pandoracalonosce:Calorimetry                   1.71691       3.34423       5.57634       3.18811       1.11258        10     
reco:pandorapid:Chi2ParticleID                    0.00451346     0.0053644    0.00621569    0.00530598    0.000592935      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0329099     0.0461952     0.0559329     0.0463386    0.00727837       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0043921    0.00500755    0.00622841    0.00480028    0.000572399      10     
reco:pandoraShowercalo:ShowerCalorimetry            3.59347       8.31675       14.564        7.68583       3.5167         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.31356       8.00713       14.8855       7.17545       3.7099         10     
reco:emtrkmichelid:EmTrackMichelId                  131.806       176.641       250.519       168.092       33.7303        10     
reco:crttag:SingleCRTMatchingProducer              0.0140819     0.0218369     0.0349144     0.0202489    0.00579547       10     
reco:crtreco:TwoCRTMatchingProducer               0.00865728     0.0166876     0.0259306     0.0157601    0.00540991       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.613291      0.918901       1.47767      0.845388      0.229112        10     
reco:pandora2Track:LArPandoraTrackCreation          11.6211       22.647        42.2572       20.5258       9.45993        10     
reco:pandora2calo:Calorimetry                       4.45917       10.6056       17.2403       10.2501       4.08529        10     
reco:pandora2calonosce:Calorimetry                  4.28906       9.90306       15.9622       9.51713       3.82536        10     
reco:pandora2pid:Chi2ParticleID                    0.0104702     0.0125109     0.0158629     0.0120075    0.00176852       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0431246     0.060347      0.0746468     0.0601953     0.0105127       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0102943     0.0126453     0.0161339     0.0116565    0.00215753       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.14845       7.24991       11.1699       7.00072       2.32975        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.80196       7.92024       12.3355       8.22392       2.58075        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.17615       6.56151       9.7088        6.84475       2.02783        10     
[art]:TriggerResults:TriggerResultInserter        3.2959e-05    4.9205e-05    0.000128483   4.14525e-05   2.68067e-05      10     
end_path:out1:RootOutput                           6.55e-06     7.5002e-06    1.1787e-05     6.955e-06    1.50571e-06      10     
end_path:out1:RootOutput(write)                     2.15572       2.91368       3.59952       2.86333       0.4744         10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5234.81 MB
  Peak resident set size usage (VmHWM): 4432.2 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.23 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.06 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11500436
-rw-r--r--. 1 dune009 dune 2652439340 Oct 29 03:12 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2517984815 Oct 29 03:10 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2376285485 Oct 29 04:20 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2120316290 Oct 29 00:52 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2025132440 Oct 29 00:39 g4_stage1.root
-rw-r--r--. 1 dune009 dune   83604640 Oct 29 04:13 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243182 Oct 29 04:20 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 29 04:20 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 29 04:20 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 29 04:20 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 29 03:12 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 29 00:52 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 29 04:20 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 29 00:32 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 29 04:20 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 29 03:18 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 29 03:12 debugprod.log
justIN time: 2025-11-03 18:43:34 UTC       justIN version: 01.05.01