justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20691.0@dunegpschedd01.fnal.gov

Jobsub ID20691.0@dunegpschedd01.fnal.gov
Workflow ID276
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-03 20:18:57
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-08-03 23:32:40
From worker nodeHostnamewn-da-14.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-03 20:23:45
Input filesmonte-carlo-000276-000001
JobscriptExit code0
Real time3h (11253s)
CPU time3h (11133s = 98%)
Max RSS bytes4007751680 (3822 MiB)
Outputting started2025-08-03 23:31:19
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00276/1/001/reco_stage1.root
Finished2025-08-03 23:32:40
Saved logsjustin-logs:20691.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce:prepare:   Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 127979 traces, 0 summary
Retagger: tagging trace set: gauss with 95845 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 95845 traces tagged "gauss"
FrameSaver: q=3.36389e+06 n=1661705 tag=gauss
wclsFrameSaver: saving 127979 traces tagged "wiener"
FrameSaver: q=3.59308e+06 n=1474173 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 14333 XUs and 22972 XVs -> 555 XUVs
C:0 T:1 69206 XUs and 151944 XVs -> 25168 XUVs
C:0 T:2 66429 XUs and 74769 XVs -> 3399 XUVs
C:0 T:3 31001 XUs and 30133 XVs -> 161 XUVs
C:0 T:4 13846 XUs and 24630 XVs -> 292 XUVs
C:0 T:5 120093 XUs and 324521 XVs -> 46037 XUVs
C:0 T:6 22120 XUs and 31291 XVs -> 898 XUVs
C:0 T:8 11465 XUs and 13671 XVs -> 280 XUVs
C:0 T:9 75260 XUs and 133235 XVs -> 9988 XUVs
C:0 T:10 22312 XUs and 13692 XVs -> 2764 XUVs
C:0 T:11 147 XUs and 87 XVs -> 14 XUVs
89556 XUVs total
11392 collection wire objects
84292 potential space points
Neighbour search...
29028570 tests to find 11035148 neighbours
Iterating with no regularization...
Begin: 7.66191e+07
0 6.27359e+07
1 6.18962e+07
2 6.17751e+07
3 6.17382e+07
Now with regularization...
Begin: 5.1353e+07
0 5.12936e+07
1 5.12547e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408442974
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408442974
fLifetime: 30000 [us]
04-Aug-2025 01:31:17 CEST  Closed output file "reco_stage1.root"
04-Aug-2025 01:31:17 CEST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          209.288       331.302       517.839       300.891       87.7006        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000863243   0.00338192    0.00592812    0.00329681    0.00142274       10     
reco:ophit:OpHitFinder                             0.192705      0.263408      0.302434      0.269739      0.0345246       10     
reco:opflash:OpFlashFinder                         0.0850798     0.145476       0.1946       0.149455      0.0317192       10     
reco:opslicer:OpSlicer                             0.388807      0.644659      0.871372      0.660692      0.130346        10     
reco:caldata:DataPrepModule                         5.03137       5.20929       5.60754       5.15663      0.183714        10     
reco:wclsdatasp:WireCellToolkit                     35.1847       48.5553       55.0847       52.7216       6.72686        10     
reco:gaushit:GausHitFinder                         0.684327       1.12379       1.8072        1.07107      0.304594        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00029952    0.000483884   0.000639397   0.000464884   0.000110494      10     
reco:reco3d:SpacePointSolver                        4.17296       11.1268       25.2237       9.23994       6.58366        10     
reco:hitpdune:DisambigFromSpacePoints              0.501382       1.18258       2.19416       1.02495      0.496087        10     
reco:pandora:StandardPandora                        10.2984       23.7212       48.1081       21.1987       11.2051        10     
reco:pandoraWriter:StandardPandora                 0.216221      0.358072      0.554833      0.341362      0.0907928       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.55002       3.15536       6.11589       2.7245        1.29328        10     
reco:pandoraShower:LArPandoraShowerCreation        0.953619       3.76727       8.72009       2.75643       2.32045        10     
reco:pandoracalo:Calorimetry                       0.759708       1.65419       3.32045       1.39921      0.726757        10     
reco:pandoracalonosce:Calorimetry                  0.650814       1.48111       3.03629       1.23956      0.675156        10     
reco:pandorapid:Chi2ParticleID                    0.00372827    0.00517514     0.0068201     0.0048782    0.000958039      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0303282     0.0446244     0.0688818     0.0429534     0.0105352       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00362919    0.00489848    0.00658673     0.0048215    0.000777974      10     
reco:pandoraShowercalo:ShowerCalorimetry           0.919254       3.46208       8.56322       2.37742       2.33424        10     
reco:pandoraShowercalonosce:ShowerCalorimetry      0.869956       3.47078       7.88077       2.31434       2.30267        10     
reco:emtrkmichelid:EmTrackMichelId                  116.452       186.802       284.785       181.886       45.0333        10     
reco:crttag:SingleCRTMatchingProducer             0.00567885     0.0105598     0.0163849    0.00950187    0.00367745       10     
reco:crtreco:TwoCRTMatchingProducer               0.00461778    0.00903004     0.0160766    0.00850788    0.00318631       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.444924      0.750067       1.00905      0.718089      0.162205        10     
reco:pandora2Track:LArPandoraTrackCreation          3.10342       10.4909       23.2137       8.46047       5.83914        10     
reco:pandora2calo:Calorimetry                       1.57574       4.73951       10.8695       3.69386       2.7813         10     
reco:pandora2calonosce:Calorimetry                  1.46378       4.56268       10.2287       3.50877       2.64352        10     
reco:pandora2pid:Chi2ParticleID                   0.00751577     0.0129267     0.0180314     0.0125975    0.00299426       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0353814     0.0565135     0.0850931     0.0552116     0.0134258       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00741429     0.0127637     0.0179578     0.0123779    0.00298385       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.21445       3.98604       8.70375       3.28268       2.19147        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.28349       4.04687       9.11808       3.13935       2.32556        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.18776       3.69415       8.21976       2.88282       2.08064        10     
[art]:TriggerResults:TriggerResultInserter         3.153e-05    4.43734e-05   0.000107705   3.6845e-05    2.16438e-05      10     
end_path:out1:RootOutput                           6.924e-06    9.2294e-06     2.111e-05     7.609e-06    4.13164e-06      10     
end_path:out1:RootOutput(write)                      1.592        2.63156       4.04222       2.58399      0.659266        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4806.1 MB
  Peak resident set size usage (VmHWM): 4007.75 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.25 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9088460
-rw-r--r--. 1 dune003 dune 2173024718 Aug  4 00:34 detsim_stage2.root
-rw-r--r--. 1 dune003 dune 2067540566 Aug  4 00:32 detsim_stage1.root
-rw-r--r--. 1 dune003 dune 1811427669 Aug  4 01:31 reco_stage1.root
-rw-r--r--. 1 dune003 dune 1632290526 Aug  3 22:41 g4_stage2.root
-rw-r--r--. 1 dune003 dune 1558873933 Aug  3 22:29 g4_stage1.root
-rw-r--r--. 1 dune003 dune   62809250 Aug  4 01:26 Pandora_Events.pndr
-rw-r--r--. 1 dune003 dune     243284 Aug  4 01:31 jobscript.log
-rw-r--r--. 1 dune003 dune     217088 Aug  4 01:31 mem.db
-rw-r--r--. 1 dune003 dune      62376 Aug  4 01:31 Pandora_Geometry.xml
-rw-r--r--. 1 dune003 dune      36864 Aug  4 01:31 time.db
-rw-r--r--. 1 dune003 dune        519 Aug  4 00:34 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 22:41 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  4 01:31 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune003 dune         52 Aug  3 22:23 all-input-dids.txt
-rw-r--r--. 1 dune003 dune          7 Aug  4 01:31 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune          0 Aug  4 00:39 ProtoDUNE.log
-rw-r--r--. 1 dune003 dune          0 Aug  4 00:34 debugprod.log
justIN time: 2025-08-04 14:17:53 UTC       justIN version: 01.04.00