justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235989.0@dunegpschedd02.fnal.gov

Jobsub ID235989.0@dunegpschedd02.fnal.gov
Workflow ID9506
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-01 09:36:05
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-01 14:58:11
From worker nodeHostnamewn-la-19.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-01 09:36:55
Input filesmonte-carlo-009506-000001
JobscriptExit code0
Real time5h (19178s)
CPU time5h (19017s = 99%)
Max RSS bytes4592652288 (4379 MiB)
Outputting started2025-11-01 14:56:33
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09506/1/001/reco_stage1.root
Finished2025-11-01 14:58:11
Saved logsjustin-logs:235989.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 180653 traces, 0 summary
Retagger: tagging trace set: gauss with 137776 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 137776 traces tagged "gauss"
FrameSaver: q=5.07741e+06 n=2510460 tag=gauss
wclsFrameSaver: saving 180653 traces tagged "wiener"
FrameSaver: q=5.38114e+06 n=2227540 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 46516 XUs and 42923 XVs -> 428 XUVs
C:0 T:1 116794 XUs and 113385 XVs -> 6876 XUVs
C:0 T:2 174793 XUs and 200634 XVs -> 10007 XUVs
C:0 T:3 71888 XUs and 74643 XVs -> 761 XUVs
C:0 T:4 19075 XUs and 16490 XVs -> 161 XUVs
C:0 T:5 89231 XUs and 83398 XVs -> 6402 XUVs
C:0 T:6 126320 XUs and 247077 XVs -> 12731 XUVs
C:0 T:7 41884 XUs and 54427 XVs -> 356 XUVs
C:0 T:8 9588 XUs and 6730 XVs -> 110 XUVs
C:0 T:9 55965 XUs and 58024 XVs -> 3527 XUVs
C:0 T:10 116275 XUs and 87887 XVs -> 15424 XUVs
C:0 T:11 472 XUs and 348 XVs -> 38 XUVs
56821 XUVs total
18790 collection wire objects
51931 potential space points
Neighbour search...
5309689 tests to find 2490276 neighbours
Iterating with no regularization...
Begin: 2.14877e+08
0 1.6066e+08
1 1.5787e+08
2 1.5756e+08
3 1.57483e+08
Now with regularization...
Begin: 1.36002e+08
0 1.35769e+08
1 1.35715e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410245116
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410245116
fLifetime: 30000 [us]
01-Nov-2025 15:56:31 CET  Closed output file "reco_stage1.root"
01-Nov-2025 15:56:31 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          392.428       578.046       773.369       560.231       133.889        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00116514    0.00582667     0.0100001    0.00583248    0.00282332       10     
reco:ophit:OpHitFinder                             0.374912      0.443172      0.536914      0.442616      0.048165        10     
reco:opflash:OpFlashFinder                         0.236445      0.319022      0.384073      0.323247      0.0490313       10     
reco:opslicer:OpSlicer                             0.671023      0.903216       1.12616       0.89076      0.152533        10     
reco:caldata:DataPrepModule                         6.26145       6.49478       7.08209       6.46172      0.217601        10     
reco:wclsdatasp:WireCellToolkit                     42.8475       54.6583       60.7013       57.8468       6.2259         10     
reco:gaushit:GausHitFinder                          1.24598       1.71019       2.13909       1.69825      0.321054        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000416781   0.000848119   0.00208976    0.000706969   0.000444042      10     
reco:reco3d:SpacePointSolver                        10.4797       27.4959       60.8427       22.3082       13.4856        10     
reco:hitpdune:DisambigFromSpacePoints               1.39875       2.47731       3.80195       2.5035       0.824934        10     
reco:pandora:StandardPandora                        31.644        59.9699       99.8425       57.8019       24.047         10     
reco:pandoraWriter:StandardPandora                  0.42361       0.58844       0.73167      0.576803      0.100715        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.01782       9.48818       13.0694        9.126        2.26963        10     
reco:pandoraShower:LArPandoraShowerCreation         5.54474       11.7289       18.0737       11.7173       4.56741        10     
reco:pandoracalo:Calorimetry                        2.90394       5.19794       7.24686       4.98443       1.23261        10     
reco:pandoracalonosce:Calorimetry                   2.62882       4.98429       8.24007       4.93538       1.42661        10     
reco:pandorapid:Chi2ParticleID                    0.00570804    0.00743266     0.0096145    0.00732452    0.00101691       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0500346     0.0659661     0.0803488     0.0657656     0.0108535       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00576129     0.0073425    0.00824541    0.00746587    0.000747251      10     
reco:pandoraShowercalo:ShowerCalorimetry            6.01812       11.915        18.5254       12.459        4.4905         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.79936       11.6958       18.7956       11.9795       4.06406        10     
reco:emtrkmichelid:EmTrackMichelId                  197.166       265.447       329.391       261.371       46.9074        10     
reco:crttag:SingleCRTMatchingProducer              0.0134496     0.0241246     0.0368142     0.0233832    0.00619099       10     
reco:crtreco:TwoCRTMatchingProducer                0.0129367     0.0174763     0.0254567     0.0176614    0.00366872       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.908279       1.23122       1.64187       1.25741      0.214818        10     
reco:pandora2Track:LArPandoraTrackCreation          16.3944       32.4029       48.491        29.8519       12.0522        10     
reco:pandora2calo:Calorimetry                       8.32328       15.7054       23.2063       15.2868       5.52484        10     
reco:pandora2calonosce:Calorimetry                  7.96087       15.0804       22.6118       14.5652       5.38685        10     
reco:pandora2pid:Chi2ParticleID                    0.0138104     0.0179202     0.0214491     0.0183357    0.00269671       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0641496     0.0871239     0.106926      0.0883067     0.0148902       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0139285     0.0179878     0.0211564     0.0184758    0.00267954       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.11464       10.6379       17.0396       10.2823       3.78614        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.65006       12.4513       18.8001       12.5943       4.1497         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.83781       10.4067       14.6995       10.3465       3.17644        10     
[art]:TriggerResults:TriggerResultInserter        4.2814e-05     6.17e-05     0.000133408   5.2878e-05    2.5229e-05       10     
end_path:out1:RootOutput                           7.621e-06    8.9323e-06    1.4251e-05    8.1365e-06    1.91333e-06      10     
end_path:out1:RootOutput(write)                     2.9973        4.13536       5.3251        4.07857      0.762056        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5398.88 MB
  Peak resident set size usage (VmHWM): 4592.65 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.10 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11677004
-rw-r--r--. 1 dune009 dune 2687719922 Nov  1 14:17 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2554884155 Nov  1 14:15 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2442064854 Nov  1 15:56 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2126067965 Nov  1 11:02 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2054440270 Nov  1 10:45 g4_stage1.root
-rw-r--r--. 1 dune009 dune   91458090 Nov  1 15:47 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243497 Nov  1 15:56 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  1 15:56 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  1 15:56 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  1 15:56 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  1 14:17 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  1 11:02 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  1 15:56 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  1 10:37 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  1 15:56 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  1 14:29 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  1 14:17 debugprod.log
justIN time: 2025-11-03 18:41:12 UTC       justIN version: 01.05.01