justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239365.1@dunegpschedd01.fnal.gov

Jobsub ID239365.1@dunegpschedd01.fnal.gov
Workflow ID9416
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-28 15:34:19
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-10-28 20:05:28
From worker nodeHostnamewn-da-23.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 15:36:19
Input filesmonte-carlo-009416-000001
JobscriptExit code0
Real time4h (16050s)
CPU time4h (15900s = 99%)
Max RSS bytes4523159552 (4313 MiB)
Outputting started2025-10-28 20:03:50
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09416/1/001/reco_stage1.root
Finished2025-10-28 20:05:28
Saved logsjustin-logs:239365.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

RawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 148413 traces, 0 summary
Retagger: tagging trace set: gauss with 113782 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 113782 traces tagged "gauss"
FrameSaver: q=3.93013e+06 n=2058534 tag=gauss
wclsFrameSaver: saving 148413 traces tagged "wiener"
FrameSaver: q=4.19188e+06 n=1842380 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 27034 XUs and 31751 XVs -> 319 XUVs
C:0 T:1 46864 XUs and 49485 XVs -> 2960 XUVs
C:0 T:2 177745 XUs and 165751 XVs -> 10147 XUVs
C:0 T:3 74508 XUs and 64180 XVs -> 703 XUVs
C:0 T:4 11941 XUs and 15558 XVs -> 112 XUVs
C:0 T:5 43668 XUs and 41111 XVs -> 3204 XUVs
C:0 T:6 205515 XUs and 164530 XVs -> 17345 XUVs
C:0 T:7 57950 XUs and 48962 XVs -> 765 XUVs
C:0 T:8 11763 XUs and 9819 XVs -> 147 XUVs
C:0 T:9 76197 XUs and 85744 XVs -> 6276 XUVs
C:0 T:10 65497 XUs and 23399 XVs -> 6983 XUVs
C:0 T:11 312 XUs and 195 XVs -> 27 XUVs
48988 XUVs total
13863 collection wire objects
44724 potential space points
Neighbour search...
4622382 tests to find 2297144 neighbours
Iterating with no regularization...
Begin: 6.16888e+07
0 5.05416e+07
1 4.96074e+07
2 4.94491e+07
3 4.93962e+07
4 4.93712e+07
Now with regularization...
Begin: 3.85421e+07
0 3.85043e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410169624
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410169624
fLifetime: 30000 [us]
28-Oct-2025 21:03:48 CET  Closed output file "reco_stage1.root"
28-Oct-2025 21:03:48 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          383.764       537.867       899.619       518.107       142.915        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00104842    0.00537417    0.00977777     0.0050708    0.00272239       10     
reco:ophit:OpHitFinder                             0.306769      0.344706      0.436469      0.333827      0.034354        10     
reco:opflash:OpFlashFinder                         0.199185      0.245995       0.3438       0.234902      0.0371636       10     
reco:opslicer:OpSlicer                             0.669675      0.867959       1.2729       0.821635      0.160878        10     
reco:caldata:DataPrepModule                         5.23432       5.41558       5.78498       5.40162      0.146435        10     
reco:wclsdatasp:WireCellToolkit                     39.5551       56.5558       80.0743       56.041        14.4675        10     
reco:gaushit:GausHitFinder                          1.25038       1.59815       2.29236       1.58075      0.282742        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000507077   0.000854201   0.00128854    0.000816218   0.000205735      10     
reco:reco3d:SpacePointSolver                        10.4139       22.102        39.0788       20.6114       7.83313        10     
reco:hitpdune:DisambigFromSpacePoints               1.38762       2.13361       3.72396       2.09786      0.625241        10     
reco:pandora:StandardPandora                        28.5693       59.1719       155.598       47.9445       35.5322        10     
reco:pandoraWriter:StandardPandora                 0.406166      0.537473      0.732055      0.522944      0.0906174       10     
reco:pandoraTrack:LArPandoraTrackCreation           5.10475       7.5721        12.238        6.73881       2.22936        10     
reco:pandoraShower:LArPandoraShowerCreation         5.14768       9.92636       23.2474       8.97229       4.88392        10     
reco:pandoracalo:Calorimetry                        2.72668       3.91597       6.2245        3.50435       1.0416         10     
reco:pandoracalonosce:Calorimetry                   2.60202       3.63057       5.77382       3.15703       0.97713        10     
reco:pandorapid:Chi2ParticleID                    0.00620601    0.00757955     0.009551     0.00702613    0.00108725       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.049491      0.0646106     0.0887828     0.0614788     0.0120573       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00584527     0.0071056    0.00901531    0.00683786    0.00105596       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.89038       9.26575       19.3027       8.79096       3.90334        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.74628       8.95354       18.9863       7.97097       3.87386        10     
reco:emtrkmichelid:EmTrackMichelId                  203.259       264.89        367.088       260.831       44.4713        10     
reco:crttag:SingleCRTMatchingProducer              0.0117326     0.0199031     0.0284328     0.0206298    0.00582505       10     
reco:crtreco:TwoCRTMatchingProducer               0.00979294     0.0157696     0.0260753     0.0155707    0.00491681       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.854427       1.09774       1.45172       1.09094      0.190735        10     
reco:pandora2Track:LArPandoraTrackCreation          13.9562       25.5174       51.9594       23.4696       10.5355        10     
reco:pandora2calo:Calorimetry                       6.48855       11.7781       22.7827       10.7452       4.44979        10     
reco:pandora2calonosce:Calorimetry                  5.89081       10.9433       21.7896       10.1972       4.40268        10     
reco:pandora2pid:Chi2ParticleID                    0.0146115     0.0187239     0.0285391     0.017978     0.00388003       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0624381     0.0814655     0.113396      0.0789434     0.0136617       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0145229     0.018807      0.0277063     0.018082     0.00381433       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.34284       9.2963        20.7885       8.94486       4.45272        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.09681       9.47367       17.7332       8.94863       3.46168        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       4.594        8.39633       15.4403       8.29232       3.03407        10     
[art]:TriggerResults:TriggerResultInserter        3.2683e-05    4.86669e-05   0.00014025    3.95815e-05   3.07769e-05      10     
end_path:out1:RootOutput                           6.372e-06    8.8522e-06    1.7384e-05    7.3645e-06    3.41781e-06      10     
end_path:out1:RootOutput(write)                     2.84808       3.78432       5.25008       3.70706       0.67017        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5321.39 MB
  Peak resident set size usage (VmHWM): 4523.16 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.23 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.38 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11264108
-rw-r--r--. 1 dune009 dune 2600708300 Oct 28 19:31 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2461187602 Oct 28 19:29 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2351046412 Oct 28 21:03 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2054800606 Oct 28 16:59 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1975936139 Oct 28 16:43 g4_stage1.root
-rw-r--r--. 1 dune009 dune   89957910 Oct 28 20:57 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243188 Oct 28 21:03 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 28 21:03 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 28 21:03 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 28 21:03 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 28 19:31 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 28 16:59 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 28 21:03 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 28 16:36 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 28 21:03 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 28 19:39 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 28 19:31 debugprod.log
justIN time: 2025-11-03 18:41:12 UTC       justIN version: 01.05.01