justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233053.0@dunegpschedd02.fnal.gov

Jobsub ID233053.0@dunegpschedd02.fnal.gov
Workflow ID9356
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-26 13:20:39
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-10-26 18:42:31
From worker nodeHostnamewn-lb-08.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-26 13:23:35
Input filesmonte-carlo-009356-000001
JobscriptExit code0
Real time5h (19026s)
CPU time5h (18849s = 99%)
Max RSS bytes4494340096 (4286 MiB)
Outputting started2025-10-26 18:40:43
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09356/1/001/reco_stage1.root
Finished2025-10-26 18:42:31
Saved logsjustin-logs:233053.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 136700 traces, 0 summary
Retagger: tagging trace set: gauss with 105156 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 105156 traces tagged "gauss"
FrameSaver: q=3.53971e+06 n=1856970 tag=gauss
wclsFrameSaver: saving 136700 traces tagged "wiener"
FrameSaver: q=3.74222e+06 n=1633539 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 34816 XUs and 35943 XVs -> 703 XUVs
C:0 T:1 140439 XUs and 92048 XVs -> 19213 XUVs
C:0 T:2 90749 XUs and 100119 XVs -> 3462 XUVs
C:0 T:3 46923 XUs and 34463 XVs -> 270 XUVs
C:0 T:4 19772 XUs and 14954 XVs -> 175 XUVs
C:0 T:5 95885 XUs and 58096 XVs -> 8826 XUVs
C:0 T:6 120582 XUs and 190915 XVs -> 8013 XUVs
C:0 T:7 46061 XUs and 57481 XVs -> 406 XUVs
C:0 T:8 10877 XUs and 8256 XVs -> 121 XUVs
C:0 T:9 86241 XUs and 83926 XVs -> 5389 XUVs
C:0 T:10 13218 XUs and 48124 XVs -> 4006 XUVs
C:0 T:11 20 XUs and 12 XVs -> 0 XUVs
50584 XUVs total
12788 collection wire objects
45937 potential space points
Neighbour search...
7679257 tests to find 3356318 neighbours
Iterating with no regularization...
Begin: 6.87335e+07
0 5.17996e+07
1 5.07995e+07
2 5.06454e+07
3 5.05946e+07
4 5.05702e+07
Now with regularization...
Begin: 4.14571e+07
0 4.13928e+07
1 4.13591e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410127525
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410127525
fLifetime: 30000 [us]
26-Oct-2025 19:40:40 CET  Closed output file "reco_stage1.root"
26-Oct-2025 19:40:40 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          392.613       553.09        874.918       536.904       124.42         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00132401    0.00672196     0.011566     0.00704239    0.00321189       10     
reco:ophit:OpHitFinder                             0.402178       0.46871      0.575219      0.458874      0.0519592       10     
reco:opflash:OpFlashFinder                         0.275524      0.346904      0.469445      0.339115      0.0600935       10     
reco:opslicer:OpSlicer                             0.644966      0.946539       1.1116        1.00035      0.142315        10     
reco:caldata:DataPrepModule                         6.34217       6.54969       7.44654       6.45414        0.307         10     
reco:wclsdatasp:WireCellToolkit                     44.2003       60.6396       68.4655       63.395        8.06338        10     
reco:gaushit:GausHitFinder                          1.16769       1.63474       2.23216       1.58953       0.27887        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000702712   0.000901411   0.00125686    0.000855265   0.000158112      10     
reco:reco3d:SpacePointSolver                        11.815        25.0544       37.3493       26.1279       8.05821        10     
reco:hitpdune:DisambigFromSpacePoints               1.22584       2.35967       4.12447       2.19813      0.805031        10     
reco:pandora:StandardPandora                        27.8077       58.1062       152.984       48.6603       33.3303        10     
reco:pandoraWriter:StandardPandora                 0.464399      0.600675      0.795113      0.593001      0.0917764       10     
reco:pandoraTrack:LArPandoraTrackCreation           7.02092       9.46358       14.039        9.01609       2.37563        10     
reco:pandoraShower:LArPandoraShowerCreation         7.12643       12.1985       22.0536       11.6434       4.05346        10     
reco:pandoracalo:Calorimetry                        3.67464       4.83797       8.22668       4.35387       1.29389        10     
reco:pandoracalonosce:Calorimetry                   3.32748       4.63087       7.26562       4.06281       1.22879        10     
reco:pandorapid:Chi2ParticleID                    0.00600735    0.00721408    0.00950553     0.0066845    0.00114235       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0504394     0.0638216     0.0825136     0.0625245    0.00932786       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0061319     0.0071021    0.00972522    0.00663696    0.00104976       10     
reco:pandoraShowercalo:ShowerCalorimetry            7.33849       12.0634       23.536        10.8377       4.43623        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       6.92851        11.41        22.2828       10.1296       4.12084        10     
reco:emtrkmichelid:EmTrackMichelId                  191.446       243.204       339.616       245.212       39.1478        10     
reco:crttag:SingleCRTMatchingProducer              0.0155354     0.0241936     0.0352663     0.0220566    0.00649415       10     
reco:crtreco:TwoCRTMatchingProducer                0.0141147     0.0186107     0.0257692     0.0179009    0.00367718       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.05485       1.20557       1.37718       1.19751      0.111171        10     
reco:pandora2Track:LArPandoraTrackCreation          20.3845       32.011        57.845        27.8391       10.7385        10     
reco:pandora2calo:Calorimetry                       10.1344       15.6188       32.502        13.5267       6.14973        10     
reco:pandora2calonosce:Calorimetry                  9.54698       14.2611       29.7921       12.9691       5.52575        10     
reco:pandora2pid:Chi2ParticleID                    0.0135277     0.0165939     0.0238075     0.0155346     0.0029814       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0634269     0.0822678     0.112132      0.0794497     0.0133891       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0136289     0.0167081     0.0241065     0.0157917    0.00304472       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.79854       10.0188       16.1655       9.48096       2.41247        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.74314       11.3857       19.1361       10.6998       2.94471        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.41667       9.59739       15.919        8.9911        2.42894        10     
[art]:TriggerResults:TriggerResultInserter        4.0019e-05    5.97001e-05   0.000136024   5.0165e-05    2.78063e-05      10     
end_path:out1:RootOutput                           7.932e-06    1.18359e-05    4.028e-05     8.718e-06    9.4902e-06       10     
end_path:out1:RootOutput(write)                     3.03033       3.95029       5.18976       3.86063      0.635407        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5297.88 MB
  Peak resident set size usage (VmHWM): 4494.34 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.10 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11545536
-rw-r--r--. 1 dune009 dune 2656650218 Oct 26 18:06 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2517320370 Oct 26 18:03 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2391800092 Oct 26 19:40 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2133655593 Oct 26 14:51 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2035768272 Oct 26 14:31 g4_stage1.root
-rw-r--r--. 1 dune009 dune   86817960 Oct 26 19:34 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243360 Oct 26 19:40 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 26 19:40 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 26 19:40 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 26 19:40 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 26 18:06 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 26 14:51 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 26 19:40 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 26 14:23 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 26 19:40 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 26 18:15 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 26 18:06 debugprod.log
justIN time: 2025-11-05 23:36:28 UTC       justIN version: 01.05.01