justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.10@dunegpschedd02.fnal.gov

Jobsub ID234970.10@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:53:17
From worker nodeHostnamedunegli-7568337-0-fnpc23018.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:42
Input filesusertests:pdhd_prod_beam__231691_95_1_20251008T183406Z_gen_g4_IonScintPDExt.root_227038_25_1_20251010T161208Z_PDInt.root
JobscriptExit code0
Real time1h (4130s)
CPU time1h (4050s = 98%)
Max RSS bytes7170383872 (6838 MiB)
Outputting started2025-10-29 20:46:33
Output files
Finished2025-10-29 20:53:17
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 neighbours
Iterating with no regularization...
Begin: 3.54369e+07
0 3.33142e+07
1 3.31618e+07
2 3.31421e+07
Now with regularization...
Begin: 3.14507e+07
0 3.1445e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:42:22 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 21123!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 118372 traces, 118372 summary
Retagger: tagging trace set: gauss with 92958 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 92958 traces tagged "gauss"
FrameSaver: q=3.19728e+06 n=1543393 tag=gauss
wclsFrameSaver: saving 118372 traces tagged "wiener"
FrameSaver: q=4.14568e+06 n=1483330 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 17270 XUs and 13794 XVs -> 167 XUVs
C:0 T:1 149080 XUs and 105641 XVs -> 6369 XUVs
C:0 T:2 25499 XUs and 24196 XVs -> 991 XUVs
C:0 T:3 5 XUs and 7 XVs -> 0 XUVs
C:0 T:4 7 XUs and 7 XVs -> 0 XUVs
C:0 T:5 59202 XUs and 47519 XVs -> 3540 XUVs
C:0 T:6 83089 XUs and 67963 XVs -> 3021 XUVs
C:0 T:7 4 XUs and 4 XVs -> 0 XUVs
14088 XUVs total
3738 collection wire objects
14088 potential space points
Neighbour search...
1061494 tests to find 591418 neighbours
Iterating with no regularization...
Begin: 1.08448e+08
0 9.02633e+07
1 8.90234e+07
2 8.88566e+07
3 8.87983e+07
Now with regularization...
Begin: 8.61961e+07
0 8.61608e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:44:25 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16150!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 105312 traces, 105312 summary
Retagger: tagging trace set: gauss with 85248 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 85248 traces tagged "gauss"
FrameSaver: q=2.81961e+06 n=1370563 tag=gauss
wclsFrameSaver: saving 105312 traces tagged "wiener"
FrameSaver: q=3.69543e+06 n=1371577 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 20478 XUs and 17303 XVs -> 282 XUVs
C:0 T:1 180205 XUs and 165544 XVs -> 7300 XUVs
C:0 T:2 10951 XUs and 17776 XVs -> 559 XUVs
C:0 T:4 8 XUs and 28 XVs -> 0 XUVs
C:0 T:5 27970 XUs and 25365 XVs -> 871 XUVs
C:0 T:6 22672 XUs and 22656 XVs -> 1366 XUVs
10378 XUVs total
3956 collection wire objects
10378 potential space points
Neighbour search...
806510 tests to find 237524 neighbours
Iterating with no regularization...
Begin: 7.35713e+07
0 6.56522e+07
1 6.49917e+07
2 6.4949e+07
Now with regularization...
Begin: 6.2136e+07
0 6.21227e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:46:32 UTC  Closed output file "pdhd_prod_beam__231691_95_1_20251008T183406Z_gen_g4_IonScintPDExt.root_227038_25_1_20251010T161208Z_PDInt.root_234970_10_1_20251029T193744Z_detsim_reco1.root"
29-Oct-2025 20:46:32 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            71.5284       97.5254       123.91        94.3183       19.1509        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000395887   0.00383571    0.00759464    0.00376248    0.00205585       10     
reco:opdec:Deconvolution                             0.524838      0.663886      0.842261      0.653557      0.0960895       10     
reco:ophitspe:OpHitFinderDeco                        0.101166      0.109375      0.121648       0.10918     0.00538356       10     
reco:opflash:OpFlashFinder                           0.0539532     0.0820332     0.126667      0.0836908     0.0207737       10     
reco:opslicer:OpSlicer                                0.1166       0.191982      0.271221      0.196066      0.0462663       10     
reco:wclsdatahd:WireCellToolkit                       32.0105       32.5626       33.0609       32.4882      0.307966        10     
reco:gaushit:GausHitFinder                           0.386421      0.571633      0.720257      0.562162      0.104917        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000194733   0.000264362   0.000432858   0.000268914   6.43649e-05      10     
reco:reco3d:SpacePointSolver                         0.283884      0.732357       1.60293      0.627612      0.362891        10     
reco:hitpdune:DisambigFromSpacePoints                0.211229      0.557271      0.918415      0.522909      0.235968        10     
reco:pandora:StandardPandora                          10.6554       17.7695       31.5624       17.0052       6.8214         10     
reco:pandoraWriter:StandardPandora                   0.134054      0.169926      0.210011      0.166085      0.0297627       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.728981       1.29868       2.00105       1.23721      0.429266        10     
reco:pandoraShower:LArPandoraShowerCreation          0.485715       1.36175       2.61707       1.20568      0.645792        10     
reco:pandoracalo:Calorimetry                         0.388918       0.74684       1.17498       0.71673      0.259863        10     
reco:pandoracalonosce:Calorimetry                    0.373161      0.739887       1.16655      0.700958      0.266015        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.486693       1.53278       3.01024       1.31555      0.792982        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.471787       1.45142       2.86427       1.25998      0.742638        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0285993     0.0379028     0.0540363     0.0362829    0.00739621       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.023458      0.035898      0.0509925     0.0345266    0.00741345       10     
reco:emtrkmichelid:EmTrackMichelId                    18.0289        23.76        29.8016       22.9775       3.80551        10     
reco:crttag:SingleCRTMatchingProducer               0.00307886    0.00400918    0.00590873    0.00389883    0.000793293      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00232002    0.00331169    0.00451943    0.00323302    0.000670069      10     
reco:pandora2Track:LArPandoraTrackCreation            1.39898       3.4463        6.15074       3.04561       1.49684        10     
reco:pandora2calo:ShowerCalorimetry                  0.423881       1.36062       2.71491       1.17401       0.71579        10     
reco:pandora2calonosce:ShowerCalorimetry             0.421992       1.33447       2.65935       1.15514      0.697892        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.025571      0.0377118     0.0516447     0.0362401    0.00737309       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0233217     0.0359973     0.0496383     0.0344357    0.00739867       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.620579       1.53059       2.73771       1.35169      0.663221        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.683819       1.82045       3.42309       1.61405      0.869092        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.674841        1.781        3.36666       1.57972      0.849537        10     
[art]:TriggerResults:TriggerResultInserter          2.2964e-05    3.48806e-05   0.00011475    2.54185e-05   2.68871e-05      10     
end_path:out1:RootOutput                             3.737e-06    9.0793e-06    3.4055e-05     7.139e-06    8.44925e-06      10     
end_path:out1:RootOutput(write)                       1.2493        1.7269        2.50857       1.72896      0.359535        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6210.44 MB
  Peak resident set size usage (VmHWM): 4007.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12301548
-rw-r--r-- 1 dunegli fnalgrid      39136 Oct 29 20:46 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Oct 29 20:46 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:46 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:46 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:46 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:46 time.db
-rw-r--r-- 1 dunegli fnalgrid 6459146356 Oct 29 20:46 pdhd_prod_beam__231691_95_1_20251008T183406Z_gen_g4_IonScintPDExt.root_227038_25_1_20251010T161208Z_PDInt.root_234970_10_1_20251029T193744Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   58206410 Oct 29 20:45 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:30 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:29 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:28 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6079125036 Oct 29 20:28 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-04 01:31:19 UTC       justIN version: 01.05.01