justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264650.1@dunegpschedd02.fnal.gov

Jobsub ID264650.1@dunegpschedd02.fnal.gov
Workflow ID11370
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-28 08:47:45
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2025-12-28 11:26:27
From worker nodeHostnamewn-s04-05-09-02.cr.cnaf.infn.it
cpuinfoAMD EPYC 7313 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-28 08:49:54
Input filesmonte-carlo-011370-000001
JobscriptExit code0
Real time2h (9085s)
CPU time2h (8761s = 96%)
Max RSS bytes4518703104 (4309 MiB)
Outputting started2025-12-28 11:21:20
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11370/1/001/reco_stage1.root
Finished2025-12-28 11:26:27
Saved logsjustin-logs:264650.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

repService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 120208 traces, 0 summary
Retagger: tagging trace set: gauss with 93148 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 93148 traces tagged "gauss"
FrameSaver: q=3.18613e+06 n=1655003 tag=gauss
wclsFrameSaver: saving 120208 traces tagged "wiener"
FrameSaver: q=3.33834e+06 n=1463520 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 29628 XUs and 30847 XVs -> 235 XUVs
C:0 T:1 64796 XUs and 62679 XVs -> 4096 XUVs
C:0 T:2 67497 XUs and 103616 XVs -> 3437 XUVs
C:0 T:3 33725 XUs and 44011 XVs -> 120 XUVs
C:0 T:4 11378 XUs and 10197 XVs -> 94 XUVs
C:0 T:5 47083 XUs and 28355 XVs -> 3698 XUVs
C:0 T:6 56332 XUs and 68356 XVs -> 2517 XUVs
C:0 T:7 26651 XUs and 29197 XVs -> 184 XUVs
C:0 T:8 9011 XUs and 5622 XVs -> 83 XUVs
C:0 T:9 46109 XUs and 41845 XVs -> 2486 XUVs
C:0 T:10 40157 XUs and 32452 XVs -> 5004 XUVs
C:0 T:11 336 XUs and 193 XVs -> 39 XUVs
21993 XUVs total
10445 collection wire objects
20424 potential space points
Neighbour search...
791048 tests to find 431598 neighbours
Iterating with no regularization...
Begin: 6.98445e+07
0 6.10853e+07
1 6.04955e+07
2 6.04016e+07
3 6.03728e+07
Now with regularization...
Begin: 5.12226e+07
0 5.1195e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411391147
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411391147
fLifetime: 30000 [us]
28-Dec-2025 12:21:18 CET  Closed output file "reco_stage1.root"
28-Dec-2025 12:21:18 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          178.991       250.848       325.722       270.089       44.0016        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000840641   0.00296806    0.00537423    0.00284798    0.00139057       10     
reco:ophit:OpHitFinder                             0.205217       0.23297      0.273087      0.235965      0.0189362       10     
reco:opflash:OpFlashFinder                         0.141694      0.169306      0.216519      0.166058      0.0222116       10     
reco:opslicer:OpSlicer                             0.377485      0.477648      0.608664      0.477755      0.0675326       10     
reco:caldata:DataPrepModule                         3.51672       3.66512       3.98493       3.65462      0.126896        10     
reco:wclsdatasp:WireCellToolkit                     23.0378       24.9388       29.4338       24.4883       1.70491        10     
reco:gaushit:GausHitFinder                         0.670927      0.895792        1.111        0.90588      0.136938        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000283974    0.000444     0.000622431   0.000424689   9.7539e-05       10     
reco:reco3d:SpacePointSolver                        3.93216       13.4279       22.2725       14.2976       5.61448        10     
reco:hitpdune:DisambigFromSpacePoints              0.813272       1.31017       1.91169       1.35721      0.315745        10     
reco:pandora:StandardPandora                        12.8727       26.2375       40.424        26.9714       8.48098        10     
reco:pandoraWriter:StandardPandora                  0.2357        0.33108      0.498605      0.328133      0.0739014       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.6633        2.82848       4.03039       2.72163       0.78955        10     
reco:pandoraShower:LArPandoraShowerCreation         1.72079       3.12611       4.7073        3.25684      0.855539        10     
reco:pandoracalo:Calorimetry                        1.03926       1.66476       2.18409       1.58514       0.39365        10     
reco:pandoracalonosce:Calorimetry                  0.948163       1.52859       2.00569       1.54152       0.34401        10     
reco:pandorapid:Chi2ParticleID                    0.00305278    0.00450333    0.00707859     0.0040875    0.00122704       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0279491     0.0402734     0.0679333     0.0397055     0.0108212       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0029751    0.00382738    0.00470796    0.00385608    0.000591247      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.94871       3.5742        5.2227        3.54241       1.00309        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.91256       3.39001       5.13222       3.46367      0.911151        10     
reco:emtrkmichelid:EmTrackMichelId                  101.594       129.835       161.087       135.785       17.7143        10     
reco:crttag:SingleCRTMatchingProducer             0.00719237     0.0116398     0.0189721     0.0101162    0.00410939       10     
reco:crtreco:TwoCRTMatchingProducer               0.00417655    0.00627879     0.010699     0.00590033    0.00170109       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.508376      0.702415       0.95601      0.724466       0.12817        10     
reco:pandora2Track:LArPandoraTrackCreation          5.04434       9.40467       16.1047       9.84199       2.95937        10     
reco:pandora2calo:Calorimetry                       2.81459       4.9348        7.5834        5.15789       1.29931        10     
reco:pandora2calonosce:Calorimetry                  2.69424       4.63617       6.59025       4.82429       1.1322         10     
reco:pandora2pid:Chi2ParticleID                    0.0063338    0.00924267     0.0113858     0.0095077     0.0013497       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0330174     0.0469666     0.0590633     0.0479582    0.00795167       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00619568    0.00909171     0.0108845    0.00956198    0.00132731       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.96246       3.19712       4.45747       3.3647        0.71954        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.55334       4.15037       5.75661       4.30826      0.913526        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.38775       3.77805       5.18187       3.94028      0.799287        10     
[art]:TriggerResults:TriggerResultInserter         3.214e-05    5.12092e-05   0.00017021    3.7991e-05    3.99556e-05      10     
end_path:out1:RootOutput                           6.923e-06    1.21638e-05   2.4055e-05    1.15315e-05   4.33014e-06      10     
end_path:out1:RootOutput(write)                     1.64619       2.19577       2.8268        2.27613      0.327872        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5323.25 MB
  Peak resident set size usage (VmHWM): 4518.7 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.25 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11004332
-rw-r--r-- 1 neutrino002 neutrino 2556298587 Dec 28 11:38 detsim_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 2424476651 Dec 28 11:36 detsim_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 2289502867 Dec 28 12:21 reco_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 1994124614 Dec 28 10:04 g4_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 1917549035 Dec 28 09:56 g4_stage1.root
-rw-r--r-- 1 neutrino002 neutrino   85856010 Dec 28 12:19 Pandora_Events.pndr
-rw-r--r-- 1 neutrino002 neutrino     242922 Dec 28 12:21 jobscript.log
-rw-r--r-- 1 neutrino002 neutrino     217088 Dec 28 12:21 mem.db
-rw-r--r-- 1 neutrino002 neutrino      62376 Dec 28 12:21 Pandora_Geometry.xml
-rw-r--r-- 1 neutrino002 neutrino      36864 Dec 28 12:21 time.db
-rw-r--r-- 1 neutrino002 neutrino        519 Dec 28 11:38 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Dec 28 10:04 g4_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Dec 28 12:21 protoDUNE_reco1_hist.root
-rw-r--r-- 1 neutrino002 neutrino         52 Dec 28 09:49 all-input-dids.txt
-rw-r--r-- 1 neutrino002 neutrino          7 Dec 28 12:21 justin-processed-pfns.txt
-rw-r--r-- 1 neutrino002 neutrino          0 Dec 28 11:42 ProtoDUNE.log
-rw-r--r-- 1 neutrino002 neutrino          0 Dec 28 11:38 debugprod.log
justIN time: 2026-02-07 03:35:44 UTC       justIN version: 01.06.00