justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264593.0@dunegpschedd02.fnal.gov

Jobsub ID264593.0@dunegpschedd02.fnal.gov
Workflow ID11361
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-27 16:54:55
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-12-27 20:21:50
From worker nodeHostnamesdsc-56.t2.ucsd.edu
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-27 16:59:00
Input filesmonte-carlo-011361-000001
JobscriptExit code0
Real time3h (12123s)
CPU time3h (11738s = 96%)
Max RSS bytes4263526400 (4066 MiB)
Outputting started2025-12-27 20:21:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11361/1/001/reco_stage1.root
Finished2025-12-27 20:21:50
Saved logsjustin-logs:264593.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 150374 traces, 0 summary
Retagger: tagging trace set: gauss with 110185 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 110185 traces tagged "gauss"
FrameSaver: q=3.70636e+06 n=1908036 tag=gauss
wclsFrameSaver: saving 150374 traces tagged "wiener"
FrameSaver: q=3.94151e+06 n=1662150 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 46778 XUs and 49120 XVs -> 747 XUVs
C:0 T:1 109406 XUs and 122987 XVs -> 9572 XUVs
C:0 T:2 60542 XUs and 101633 XVs -> 4557 XUVs
C:0 T:3 31069 XUs and 38305 XVs -> 142 XUVs
C:0 T:4 14437 XUs and 17707 XVs -> 236 XUVs
C:0 T:5 66003 XUs and 61247 XVs -> 6492 XUVs
C:0 T:6 110942 XUs and 92270 XVs -> 4552 XUVs
C:0 T:7 45675 XUs and 33705 XVs -> 419 XUVs
C:0 T:8 8364 XUs and 8705 XVs -> 90 XUVs
C:0 T:9 61917 XUs and 84538 XVs -> 5577 XUVs
C:0 T:10 38236 XUs and 22458 XVs -> 4114 XUVs
C:0 T:11 306 XUs and 117 XVs -> 27 XUVs
36525 XUVs total
14709 collection wire objects
32807 potential space points
Neighbour search...
1696889 tests to find 887940 neighbours
Iterating with no regularization...
Begin: 6.49227e+07
0 4.91233e+07
1 4.81084e+07
2 4.79586e+07
3 4.79143e+07
Now with regularization...
Begin: 3.71225e+07
0 3.70952e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411377764
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411377764
fLifetime: 30000 [us]
27-Dec-2025 12:21:02 PST  Closed output file "reco_stage1.root"
27-Dec-2025 12:21:02 PST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          297.748       427.317       509.831       446.421       70.0026        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000921426   0.00366053    0.00580667    0.00404942    0.00163307       10     
reco:ophit:OpHitFinder                             0.265959      0.292922      0.310483       0.29805      0.015328        10     
reco:opflash:OpFlashFinder                         0.175072       0.21443       0.24736       0.21548      0.0230938       10     
reco:opslicer:OpSlicer                             0.621715      0.730539      0.829169      0.737907      0.074776        10     
reco:caldata:DataPrepModule                         5.47026       5.69322       6.01313       5.67372      0.152595        10     
reco:wclsdatasp:WireCellToolkit                     44.195        47.2904       51.2164       47.0083       2.14687        10     
reco:gaushit:GausHitFinder                          1.07272       1.43072       1.73196       1.49235      0.220481        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000451135   0.000685081   0.000840972   0.000729082   0.000122169      10     
reco:reco3d:SpacePointSolver                        10.9652       16.6489       22.3138       16.2433       3.68765        10     
reco:hitpdune:DisambigFromSpacePoints               1.67157       2.71556       3.39623       2.90681      0.604001        10     
reco:pandora:StandardPandora                        27.1318       48.1901       62.9429       49.5589       12.5122        10     
reco:pandoraWriter:StandardPandora                 0.351467      0.496791      0.567477      0.537129      0.0783065       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.75531       6.65488       8.60054       6.76169       1.42675        10     
reco:pandoraShower:LArPandoraShowerCreation         2.97756       6.60825       9.51688       7.19871       2.16793        10     
reco:pandoracalo:Calorimetry                        1.96241       3.21791       4.28346       3.33969      0.710171        10     
reco:pandoracalonosce:Calorimetry                   1.72477       2.86741       3.9695        2.84909      0.643418        10     
reco:pandorapid:Chi2ParticleID                    0.00515789    0.00637055    0.00758281    0.00634765    0.000716091      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP          0.05442      0.0749307     0.0893072     0.0783207     0.0100205       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00501205    0.00613583    0.00734574    0.00601758    0.000633491      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.63842       6.48282       9.93101       7.04223       2.32802        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.5707        6.2984        9.22531       7.09524       2.19987        10     
reco:emtrkmichelid:EmTrackMichelId                  152.756       204.909       238.641       214.25        28.3709        10     
reco:crttag:SingleCRTMatchingProducer              0.0141363     0.0207311     0.0234165     0.0218283    0.00274015       10     
reco:crtreco:TwoCRTMatchingProducer               0.00856297     0.0137413     0.0169063     0.0145568    0.00278347       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.57276       1.93957       2.4857        1.94866      0.246023        10     
reco:pandora2Track:LArPandoraTrackCreation          10.7115       21.664        29.3345       23.4567       6.36533        10     
reco:pandora2calo:Calorimetry                       4.42909       9.11037       12.8484       9.82465       2.74669        10     
reco:pandora2calonosce:Calorimetry                  4.23552       8.56126       12.0689       9.28316       2.56567        10     
reco:pandora2pid:Chi2ParticleID                    0.0119064     0.0159292     0.0197865     0.0164145    0.00249221       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0674882     0.0939405     0.108871      0.100429      0.0136992       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0117768     0.0157108     0.0191441     0.0158134    0.00234809       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.46203       6.87781       10.0059       7.29443       1.97273        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.85266       7.82453       10.4845       8.60902       2.25434        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.52701       7.28158       10.4874       7.98832       2.17856        10     
[art]:TriggerResults:TriggerResultInserter         4.841e-05    5.86358e-05   0.000116933   5.3006e-05    1.95531e-05      10     
end_path:out1:RootOutput                           8.299e-06    1.00896e-05   1.4687e-05    9.7235e-06    1.65587e-06      10     
end_path:out1:RootOutput(write)                     2.24125       2.9562        3.47459       3.03732      0.408684        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5076.99 MB
  Peak resident set size usage (VmHWM): 4263.53 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.57 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10546040
-rw-r--r--. 1 cuser cuser 2467789644 Dec 27 11:07 detsim_stage2.root
-rw-r--r--. 1 cuser cuser 2337846894 Dec 27 11:04 detsim_stage1.root
-rw-r--r--. 1 cuser cuser 2179670042 Dec 27 12:21 reco_stage1.root
-rw-r--r--. 1 cuser cuser 1905149007 Dec 27 09:22 g4_stage2.root
-rw-r--r--. 1 cuser cuser 1827100358 Dec 27 09:07 g4_stage1.root
-rw-r--r--. 1 cuser cuser   80972560 Dec 27 12:15 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser     242899 Dec 27 12:21 jobscript.log
-rw-r--r--. 1 cuser cuser     217088 Dec 27 12:21 mem.db
-rw-r--r--. 1 cuser cuser      62376 Dec 27 12:21 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser      36864 Dec 27 12:21 time.db
-rw-r--r--. 1 cuser cuser        519 Dec 27 11:07 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 cuser cuser        519 Dec 27 09:22 g4_protoDUNE_hist.root
-rw-r--r--. 1 cuser cuser        519 Dec 27 12:21 protoDUNE_reco1_hist.root
-rw-r--r--. 1 cuser cuser         52 Dec 27 08:59 all-input-dids.txt
-rw-r--r--. 1 cuser cuser          7 Dec 27 12:21 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser          0 Dec 27 11:13 ProtoDUNE.log
-rw-r--r--. 1 cuser cuser          0 Dec 27 11:07 debugprod.log
justIN time: 2026-02-07 03:34:51 UTC       justIN version: 01.06.00