justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.20@dunegpschedd02.fnal.gov

Jobsub ID234970.20@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 21:00:18
From worker nodeHostnamedunegli-7568338-0-fnpc23050.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:44
Input filesusertests:pdhd_prod_beam__231691_2_1_20251008T183040Z_gen_g4_IonScintPDExt.root_227038_134_1_20251010T164331Z_PDInt.root
JobscriptExit code0
Real time1h (4551s)
CPU time1h (4470s = 98%)
Max RSS bytes7620984832 (7267 MiB)
Outputting started2025-10-29 20:53:36
Output files
Finished2025-10-29 21:00:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ating with no regularization...
Begin: 5.44427e+07
0 4.87161e+07
1 4.82647e+07
2 4.82266e+07
Now with regularization...
Begin: 4.60695e+07
0 4.60591e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:49:58 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16853!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 96557 traces, 96557 summary
Retagger: tagging trace set: gauss with 77189 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77189 traces tagged "gauss"
FrameSaver: q=2.44369e+06 n=1285534 tag=gauss
wclsFrameSaver: saving 96557 traces tagged "wiener"
FrameSaver: q=2.93106e+06 n=1228449 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8316 XUs and 7726 XVs -> 45 XUVs
C:0 T:1 37555 XUs and 33881 XVs -> 771 XUVs
C:0 T:2 30177 XUs and 24236 XVs -> 686 XUVs
C:0 T:3 2 XUs and 3 XVs -> 0 XUVs
C:0 T:4 5 XUs and 12 XVs -> 0 XUVs
C:0 T:5 24826 XUs and 25552 XVs -> 555 XUVs
C:0 T:6 22917 XUs and 17382 XVs -> 800 XUVs
C:0 T:7 17 XUs and 14 XVs -> 0 XUVs
2857 XUVs total
1975 collection wire objects
2857 potential space points
Neighbour search...
45021 tests to find 23216 neighbours
Iterating with no regularization...
Begin: 3.03641e+07
0 2.8634e+07
1 2.8546e+07
2 2.85375e+07
Now with regularization...
Begin: 2.70265e+07
0 2.70237e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:51:27 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17879!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 111675 traces, 111675 summary
Retagger: tagging trace set: gauss with 91231 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 91231 traces tagged "gauss"
FrameSaver: q=2.93569e+06 n=1464591 tag=gauss
wclsFrameSaver: saving 111675 traces tagged "wiener"
FrameSaver: q=3.72699e+06 n=1413722 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 22131 XUs and 17338 XVs -> 220 XUVs
C:0 T:1 115736 XUs and 96649 XVs -> 4050 XUVs
C:0 T:2 23699 XUs and 28907 XVs -> 1344 XUVs
C:0 T:4 2 XUs and 3 XVs -> 0 XUVs
C:0 T:5 71139 XUs and 48098 XVs -> 3072 XUVs
C:0 T:6 27211 XUs and 28687 XVs -> 1074 XUVs
C:0 T:7 3 XUs and 5 XVs -> 0 XUVs
9760 XUVs total
4461 collection wire objects
9760 potential space points
Neighbour search...
468884 tests to find 168490 neighbours
Iterating with no regularization...
Begin: 5.34685e+07
0 4.64011e+07
1 4.58547e+07
2 4.58e+07
3 4.57863e+07
Now with regularization...
Begin: 4.32902e+07
0 4.32787e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:53:35 UTC  Closed output file "pdhd_prod_beam__231691_2_1_20251008T183040Z_gen_g4_IonScintPDExt.root_227038_134_1_20251010T164331Z_PDInt.root_234970_20_1_20251029T193746Z_detsim_reco1.root"
29-Oct-2025 20:53:35 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            69.4066       108.618       137.822       106.555       23.205         10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000629375   0.00469675    0.00775587    0.00487862    0.00231562       10     
reco:opdec:Deconvolution                             0.524611      0.661272      0.811245       0.65803      0.0795392       10     
reco:ophitspe:OpHitFinderDeco                        0.100007      0.108499      0.122149      0.106931     0.00597983       10     
reco:opflash:OpFlashFinder                           0.0642027     0.0903195     0.142034      0.0846227     0.0217737       10     
reco:opslicer:OpSlicer                               0.155955      0.206534      0.314679       0.19279      0.0472357       10     
reco:wclsdatahd:WireCellToolkit                       31.0758       31.9846       33.269        31.8159      0.693881        10     
reco:gaushit:GausHitFinder                           0.463008      0.636963      0.817828      0.628341      0.116627        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000179393   0.000292391   0.000525947   0.000277491   9.16038e-05      10     
reco:reco3d:SpacePointSolver                         0.199294       0.71435       1.16851      0.724708      0.302814        10     
reco:hitpdune:DisambigFromSpacePoints                0.221575      0.688181       1.1075        0.66557       0.29201        10     
reco:pandora:StandardPandora                          6.08558       22.7403       39.4435       21.2185       9.78429        10     
reco:pandoraWriter:StandardPandora                    0.13325      0.193443      0.260123      0.190862      0.0439652       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.763561       1.73386       2.62875       1.83428      0.629443        10     
reco:pandoraShower:LArPandoraShowerCreation          0.894845        1.71         2.69814       1.60359      0.619258        10     
reco:pandoracalo:Calorimetry                          0.42885      0.946448       1.37463      0.992766      0.333602        10     
reco:pandoracalonosce:Calorimetry                    0.422728      0.930322       1.34484      0.978951      0.325863        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.938961       1.83234       3.09429       1.75788      0.716448        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.887009       1.73899       2.92408       1.66874      0.680858        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0295873     0.0393768     0.0533607     0.0387027    0.00761843       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0282601     0.037623      0.0503991     0.0353594    0.00741361       10     
reco:emtrkmichelid:EmTrackMichelId                    17.8731       25.8954       33.3189       25.952        4.61609        10     
reco:crttag:SingleCRTMatchingProducer               0.00309234    0.00429151    0.00598755    0.00418865    0.00103602       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00266587    0.00346586    0.00469039    0.00325974    0.000666094      10     
reco:pandora2Track:LArPandoraTrackCreation            1.9823        4.29641       6.43912       4.24299       1.53168        10     
reco:pandora2calo:ShowerCalorimetry                   0.78005       1.62347       2.73044       1.54523      0.649963        10     
reco:pandora2calonosce:ShowerCalorimetry             0.745865       1.59482       2.66917       1.51888      0.638451        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0290153     0.039233      0.0527953     0.0373591    0.00782233       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0278001     0.0372141     0.0503365     0.0349367    0.00729203       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.892801       1.91154       2.86831       1.84311      0.662647        10     
reco:pandora2Showercalo:ShowerCalorimetry             0.98368       2.20727       3.43702       2.17478       0.84872        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.951756       2.14807       3.35661       2.10862      0.831521        10     
[art]:TriggerResults:TriggerResultInserter          1.9428e-05    3.19222e-05   9.0744e-05    2.63455e-05   1.98844e-05      10     
end_path:out1:RootOutput                             3.928e-06    7.3963e-06    2.7463e-05    4.9795e-06    6.79359e-06      10     
end_path:out1:RootOutput(write)                       1.27976       1.79466       2.1975        1.7771       0.303057        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6497.93 MB
  Peak resident set size usage (VmHWM): 4285.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 13357084
-rw-r--r-- 1 dunegli fnalgrid      39262 Oct 29 20:53 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Oct 29 20:53 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:53 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:53 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:53 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:53 time.db
-rw-r--r-- 1 dunegli fnalgrid 7016118602 Oct 29 20:53 pdhd_prod_beam__231691_2_1_20251008T183040Z_gen_g4_IonScintPDExt.root_227038_134_1_20251010T164331Z_PDInt.root_234970_20_1_20251029T193746Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   65276660 Oct 29 20:52 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:35 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:34 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:34 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6595953889 Oct 29 20:34 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-04 01:29:48 UTC       justIN version: 01.05.01