Jobsub ID 38967.28@dunegpschedd02.fnal.gov

Jobsub ID	38967.28@dunegpschedd02.fnal.gov
Workflow ID	2536
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-12 13:57:56
Site	IT_CNAF
Entry	CMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat	2025-09-12 17:10:42
From worker node	Hostname	cn-leo-176.cr.cnaf.infn.it
	cpuinfo	Intel(R) Xeon(R) Platinum 8480+
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-09-12 16:00:37
Input files	vd-protodune:np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643.hdf5
Outputting started	2025-09-12 17:06:16
Output files	vd-protodune-det-reco:np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup.root
Finished	2025-09-12 17:10:35
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

window for channel 8076
wclsFrameSaver: no samples within desired window for channel 8077
wclsFrameSaver: no samples within desired window for channel 8077
wclsFrameSaver: no samples within desired window for channel 8077
wclsFrameSaver: no samples within desired window for channel 8078
wclsFrameSaver: no samples within desired window for channel 8078
wclsFrameSaver: no samples within desired window for channel 8079
wclsFrameSaver: no samples within desired window for channel 8079
wclsFrameSaver: no samples within desired window for channel 8080
wclsFrameSaver: no samples within desired window for channel 8080
wclsFrameSaver: no samples within desired window for channel 8885
wclsFrameSaver: no samples within desired window for channel 8904
wclsFrameSaver: no samples within desired window for channel 8905
wclsFrameSaver: no samples within desired window for channel 8905
wclsFrameSaver: no samples within desired window for channel 8906
wclsFrameSaver: no samples within desired window for channel 8906
wclsFrameSaver: no samples within desired window for channel 8907
wclsFrameSaver: no samples within desired window for channel 8907
wclsFrameSaver: no samples within desired window for channel 8908
wclsFrameSaver: no samples within desired window for channel 8908
wclsFrameSaver: no samples within desired window for channel 8909
wclsFrameSaver: no samples within desired window for channel 8910
wclsFrameSaver: no samples within desired window for channel 8910
wclsFrameSaver: no samples within desired window for channel 9053
wclsFrameSaver: no samples within desired window for channel 9053
wclsFrameSaver: no samples within desired window for channel 9054
wclsFrameSaver: no samples within desired window for channel 9054
wclsFrameSaver: no samples within desired window for channel 9058
wclsFrameSaver: no samples within desired window for channel 9059
wclsFrameSaver: no samples within desired window for channel 9059
wclsFrameSaver: no samples within desired window for channel 9059
wclsFrameSaver: no samples within desired window for channel 9319
wclsFrameSaver: no samples within desired window for channel 10297
wclsFrameSaver: no samples within desired window for channel 10297
wclsFrameSaver: no samples within desired window for channel 10320
wclsFrameSaver: no samples within desired window for channel 10387
wclsFrameSaver: no samples within desired window for channel 10583
wclsFrameSaver: no samples within desired window for channel 10840
wclsFrameSaver: no samples within desired window for channel 11228
FrameSaver: q=9.82141e+06 n=1125210 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 3093 XUs and 2577 XVs -> 118 XUVs
C:0 T:1 401 XUs and 377 XVs -> 10 XUVs
C:0 T:2 3784 XUs and 4560 XVs -> 119 XUVs
C:0 T:3 165 XUs and 291 XVs -> 11 XUVs
C:0 T:4 2437 XUs and 1785 XVs -> 122 XUVs
C:0 T:5 738 XUs and 679 XVs -> 27 XUVs
C:0 T:6 1050 XUs and 478 XVs -> 48 XUVs
C:0 T:7 296 XUs and 663 XVs -> 74 XUVs
C:0 T:8 2189 XUs and 2354 XVs -> 98 XUVs
C:0 T:9 5178 XUs and 6058 XVs -> 282 XUVs
C:0 T:10 4785 XUs and 4201 XVs -> 224 XUVs
C:0 T:11 649 XUs and 882 XVs -> 55 XUVs
C:0 T:12 5937 XUs and 5145 XVs -> 396 XUVs
C:0 T:13 3095 XUs and 4120 XVs -> 228 XUVs
C:0 T:14 1314 XUs and 1035 XVs -> 65 XUVs
C:0 T:15 2665 XUs and 4541 XVs -> 409 XUVs
2286 XUVs total
1768 collection wire objects
2286 potential space points
Neighbour search...
30750 tests to find 15982 neighbours
Iterating with no regularization...
Begin: 1.01537e+09
0 9.72205e+08
1 9.705e+08
2 9.70455e+08
Now with regularization...
Begin: 9.62026e+08
0 9.62019e+08
12-Sep-2025 19:06:08 CEST  Closed output file "np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        147.664       216.156       335.551       197.296       50.7279        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.2323e-05    0.000109407   0.000211163   8.9233e-05    3.99448e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               13.6567       17.2987       19.9654       17.6148       2.17925        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0210804     0.0241219     0.0527142     0.0218124    0.00748473       18     
produce:pdvddaphne:DAPHNEReaderPDVD               4.22901       4.80317       5.69638       4.69645      0.441758        18     
produce:ophit:OpHitFinder                        0.0392643     0.0434333     0.0503639     0.0434756    0.00306737       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0080299     0.0121344     0.0177094     0.0120044    0.00258151       18     
produce:wclsdatavd:WireCellToolkit                78.3703       93.1733       112.553       93.6932       8.42793        18     
produce:gaushit:GausHitFinder                    0.846393       1.32992       2.80598       1.19208      0.412659        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000316363   0.00052286    0.000785192   0.000513058   0.000114996      18     
produce:reco3d:SpacePointSolver                   10.4158       17.9878       28.4509       17.0018       4.37285        18     
produce:hitpdune:DisambigFromSpacePoints         0.127607      0.242842       0.42298      0.235446      0.0689739       18     
produce:pandora:StandardPandora                   29.5435       75.0479       183.018       59.102        41.5209        18     
produce:pandoraTrack:LArPandoraTrackCreation      0.95628       1.67415       3.18287       1.60287       0.53714        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0198104     0.0294321     0.0431656     0.0292521    0.00707947       18     
[art]:TriggerResults:TriggerResultInserter      3.1166e-05    4.65983e-05   7.7857e-05    4.36315e-05   1.18908e-05      18     
end_path:out1:RootOutput                         7.01e-06     1.11473e-05   3.1167e-05    1.0273e-05    5.24356e-06      18     
end_path:out1:RootOutput(write)                   3.89256       4.42007       5.08947       4.43949      0.303959        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5085.32 MB
  Peak resident set size usage (VmHWM): 3183.27 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup.root
\tHists: np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757696768.0,
    "core.end_time": 1757696768.0,
    "core.events": [
      486722,
      486742,
      486762,
      486782,
      486802,
      486822,
      486842,
      486862,
      486882,
      486902,
      486922,
      486942,
      486962,
      486982,
      487002,
      487022,
      487042,
      487062
    ],
    "core.event_count": 18,
    "core.first_event_number": 486722,
    "core.last_event_number": 487062,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643_reco_stage1_20250912T170608_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1757696768.0,
    "core.end_time": 1757696768.0,
    "core.data_stream": "physics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39273
    ],
    "core.runs_subruns": [
      3927300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039273_1352_df-s02-d0_dw_0_20250901T040643.hdf5"
    }
  ]
}