justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233124.0@dunegpschedd02.fnal.gov

Jobsub ID233124.0@dunegpschedd02.fnal.gov
Workflow ID9365
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-26 22:45:11
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-10-27 02:47:06
From worker nodeHostnamewn-lb-12.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-26 22:46:48
Input filesmonte-carlo-009365-000001
JobscriptExit code0
Real time3h (14313s)
CPU time3h (14105s = 98%)
Max RSS bytes4192444416 (3998 MiB)
Outputting started2025-10-27 02:45:22
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09365/1/001/reco_stage1.root
Finished2025-10-27 02:47:06
Saved logsjustin-logs:233124.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 135326 traces, 0 summary
Retagger: tagging trace set: gauss with 103169 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 103169 traces tagged "gauss"
FrameSaver: q=3.39872e+06 n=1788671 tag=gauss
wclsFrameSaver: saving 135326 traces tagged "wiener"
FrameSaver: q=3.61153e+06 n=1581673 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 24697 XUs and 34508 XVs -> 253 XUVs
C:0 T:1 49283 XUs and 55344 XVs -> 2799 XUVs
C:0 T:2 134182 XUs and 171301 XVs -> 6419 XUVs
C:0 T:3 61163 XUs and 65740 XVs -> 632 XUVs
C:0 T:4 7436 XUs and 7329 XVs -> 165 XUVs
C:0 T:5 22842 XUs and 28757 XVs -> 5848 XUVs
C:0 T:6 184610 XUs and 186027 XVs -> 13697 XUVs
C:0 T:7 63947 XUs and 62791 XVs -> 699 XUVs
C:0 T:8 6300 XUs and 6584 XVs -> 75 XUVs
C:0 T:9 44899 XUs and 62470 XVs -> 3854 XUVs
C:0 T:10 20612 XUs and 12555 XVs -> 2227 XUVs
C:0 T:11 211 XUs and 141 XVs -> 15 XUVs
36683 XUVs total
11715 collection wire objects
33375 potential space points
Neighbour search...
2758645 tests to find 1383856 neighbours
Iterating with no regularization...
Begin: 5.50483e+07
0 4.46823e+07
1 4.39213e+07
2 4.37956e+07
3 4.37562e+07
Now with regularization...
Begin: 3.50312e+07
0 3.50143e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410134459
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410134459
fLifetime: 30000 [us]
27-Oct-2025 03:45:20 CET  Closed output file "reco_stage1.root"
27-Oct-2025 03:45:20 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          331.632       396.705       464.153       395.782       40.404         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00123768    0.00570312     0.0106689    0.00584396    0.00295697       10     
reco:ophit:OpHitFinder                             0.305106      0.378199      0.435898      0.391723      0.0436104       10     
reco:opflash:OpFlashFinder                         0.177624      0.240979      0.298631      0.247119      0.0405564       10     
reco:opslicer:OpSlicer                             0.539046      0.729325      0.882433      0.756196      0.0980776       10     
reco:caldata:DataPrepModule                         5.25549       6.27584       6.96886       6.34606      0.398619        10     
reco:wclsdatasp:WireCellToolkit                     43.9009       61.5997       71.752        64.9621       10.0149        10     
reco:gaushit:GausHitFinder                          1.21893       1.34248       1.51833       1.33495      0.0850368       10     
reco:nhitsfilter:NumberOfHitsFilter               0.000398868   0.00050782    0.000708793   0.000469156   0.00010651       10     
reco:reco3d:SpacePointSolver                        14.1915       19.3469       27.5677       18.3789       4.30879        10     
reco:hitpdune:DisambigFromSpacePoints               1.23102       1.53981       1.82868       1.51472      0.188409        10     
reco:pandora:StandardPandora                        25.3859       35.1365       50.6162       33.5308       7.20856        10     
reco:pandoraWriter:StandardPandora                 0.403775      0.476927      0.542577      0.474401      0.0446895       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.64376       6.36906       9.4364        6.1461        1.31692        10     
reco:pandoraShower:LArPandoraShowerCreation         4.55916       6.64769       9.42816       6.15215       1.43238        10     
reco:pandoracalo:Calorimetry                        2.36912       3.37129       4.94337       3.17013      0.718136        10     
reco:pandoracalonosce:Calorimetry                   2.18482       3.2115        4.78315       3.02185      0.721713        10     
reco:pandorapid:Chi2ParticleID                    0.00439542     0.006238     0.00788108    0.00625351    0.000928548      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0446404     0.0528622     0.061693      0.0529458    0.00461656       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0044955    0.00589931    0.00691629    0.00612598    0.000653966      10     
reco:pandoraShowercalo:ShowerCalorimetry            4.11685       6.6407        9.1144        6.43468       1.51173        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.70376       6.28131       8.77497       6.17841       1.46218        10     
reco:emtrkmichelid:EmTrackMichelId                  164.057       181.171       206.53        178.27        13.5005        10     
reco:crttag:SingleCRTMatchingProducer              0.0140613     0.0172857     0.0220096     0.0171043    0.00252502       10     
reco:crtreco:TwoCRTMatchingProducer               0.00998899     0.0136167     0.0180331     0.0141524    0.00227397       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.83525      0.988886       1.11245      0.991696      0.0817246       10     
reco:pandora2Track:LArPandoraTrackCreation          12.5928       17.2786       21.6763       17.2159       2.71449        10     
reco:pandora2calo:Calorimetry                       5.29693       8.31955       10.5138       8.65665       1.67134        10     
reco:pandora2calonosce:Calorimetry                  5.14079       7.79741       11.1881       8.07183       1.74848        10     
reco:pandora2pid:Chi2ParticleID                   0.00917869     0.0116963     0.0134777     0.0116996    0.00122885       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0495146     0.059476      0.0678505     0.0603347    0.00693796       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0091577     0.0117969     0.0135439     0.0118659    0.00130642       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.14604       5.73751       7.29008       5.8422       0.978844        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.41896       6.60608       8.79576       7.17934       1.47932        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.11764       5.89433       8.48987       6.19581       1.44937        10     
[art]:TriggerResults:TriggerResultInserter        3.1837e-05    5.37552e-05   0.000103735   4.6104e-05    1.98657e-05      10     
end_path:out1:RootOutput                           6.931e-06    8.2643e-06    1.1207e-05     8.032e-06    1.10458e-06      10     
end_path:out1:RootOutput(write)                     2.45067       2.91642       3.57097       2.98037      0.375525        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4995.46 MB
  Peak resident set size usage (VmHWM): 4192.44 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.91 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9848104
-rw-r--r--. 1 dune009 dune 2334602836 Oct 27 02:37 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2217292423 Oct 27 02:34 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2003508479 Oct 27 03:45 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1768868964 Oct 27 00:06 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1689288100 Oct 26 23:54 g4_stage1.root
-rw-r--r--. 1 dune009 dune   70218190 Oct 27 03:40 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242954 Oct 27 03:45 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 27 03:45 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 27 03:45 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 27 03:45 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 27 02:37 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 27 00:06 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 27 03:45 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 26 23:46 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 27 03:45 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 27 02:43 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 27 02:37 debugprod.log
justIN time: 2025-11-04 07:41:29 UTC       justIN version: 01.05.01