Jobsub ID 36378.43@dunegpschedd02.fnal.gov
| Jobsub ID | 36378.43@dunegpschedd02.fnal.gov |
| Workflow ID | 2332 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 13:34:00 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce508 |
| Last heartbeat | 2025-09-08 15:52:37 |
| From worker node | Hostname | b9p10p2030.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:57:38 |
| Input files | vd-protodune:np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 50m (3024s) |
| CPU time | 45m (2707s = 89%) |
| Max RSS bytes | 3052187648 (2910 MiB) |
| Outputting started | 2025-09-08 15:48:02 |
| Output files | |
| Finished | 2025-09-08 15:52:37 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
igproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=53 anode=0
[17:43:35.377] D [ glue ] <FrameFanin:nfsp> EOS at call=53 with 8
[17:43:35.377] D [ glue ] frame sink sees EOS
[17:43:35.377] D [ pgraph ] <Pgrapher:> graph execution complete
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.15 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.46 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.35 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.75 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.65 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.59 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.59 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.47 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Aux::Resampler : 0.23 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Aux::Resampler : 0.22 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Aux::Resampler : 0.21 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:43:35.377] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:43:35.377] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:43:35.377] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:43:35.377] I [ timer ] Timer: Total node execution : 35.92999998666346 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 34335 traces tagged "gauss"
FrameSaver: q=8.54059e+06 n=840601 tag=gauss
wclsFrameSaver: saving 43053 traces tagged "wiener"
FrameSaver: q=9.06759e+06 n=803725 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 714 XUs and 757 XVs -> 26 XUVs
C:0 T:1 239 XUs and 314 XVs -> 17 XUVs
C:0 T:2 662 XUs and 423 XVs -> 37 XUVs
C:0 T:3 438 XUs and 629 XVs -> 16 XUVs
C:0 T:4 270 XUs and 226 XVs -> 21 XUVs
C:0 T:5 204 XUs and 179 XVs -> 12 XUVs
C:0 T:6 783 XUs and 993 XVs -> 37 XUVs
C:0 T:7 48 XUs and 63 XVs -> 3 XUVs
C:0 T:8 9655 XUs and 14134 XVs -> 2784 XUVs
C:0 T:9 14430 XUs and 19400 XVs -> 2326 XUVs
C:0 T:10 11699 XUs and 21482 XVs -> 3090 XUVs
C:0 T:11 2958 XUs and 3056 XVs -> 365 XUVs
C:0 T:12 13133 XUs and 25917 XVs -> 2213 XUVs
C:0 T:13 830 XUs and 725 XVs -> 58 XUVs
C:0 T:14 587 XUs and 608 XVs -> 24 XUVs
C:0 T:15 417 XUs and 1000 XVs -> 39 XUVs
11068 XUVs total
1820 collection wire objects
11068 potential space points
Neighbour search...
1504478 tests to find 658498 neighbours
Iterating with no regularization...
Begin: 4.48531e+09
0 4.22491e+09
1 4.20841e+09
2 4.20771e+09
Now with regularization...
Begin: 4.18849e+09
0 4.18815e+09
08-Sep-2025 17:44:13 CEST Closed output file "np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357_reco_stage1_20250908T154413_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 66.5898 102.276 183.112 97.5397 27.2291 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.103e-05 7.0333e-05 0.000218071 6.3221e-05 3.27143e-05 27
produce:tpcrawdecoder:PDVDTPCReader 3.08935 4.24827 8.01133 3.8713 1.15605 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.00977724 0.0237622 0.164718 0.0115718 0.0325474 27
produce:pdvddaphne:DAPHNEReaderPDVD 1.771 2.31434 2.79785 2.32176 0.25339 27
produce:ophit:OpHitFinder 0.0261774 0.0363628 0.0476805 0.0363028 0.00525304 27
produce:opflash:OpFlashFinderVerticalDrift 0.00346096 0.00870559 0.0153766 0.00793669 0.00317857 27
produce:wclsdatavd:WireCellToolkit 32.5379 40.0115 67.6105 37.6064 9.02981 27
produce:gaushit:GausHitFinder 0.580997 1.11847 2.20744 0.940051 0.414826 27
produce:nhitsfilter:NumberOfHitsFilter 0.000164361 0.000228604 0.000725255 0.000194341 0.000105174 27
produce:reco3d:SpacePointSolver 4.82605 9.04295 18.8424 8.85687 2.86138 27
produce:hitpdune:DisambigFromSpacePoints 0.0620847 0.128728 0.276558 0.127122 0.0445821 27
produce:pandora:StandardPandora 15.6652 41.3666 107.402 37.3532 22.4714 27
produce:pandoraTrack:LArPandoraTrackCreation 0.223762 0.539442 1.12599 0.506788 0.237302 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.00991055 0.0212014 0.0412934 0.0188693 0.00743437 27
[art]:TriggerResults:TriggerResultInserter 1.463e-05 2.50821e-05 9.408e-05 2.222e-05 1.44895e-05 27
end_path:out1:RootOutput 3.99e-06 7.53444e-06 3.144e-05 7.22e-06 4.96397e-06 27
end_path:out1:RootOutput(write) 2.67017 3.37256 5.34624 3.18399 0.561 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5006.13 MB
Peak resident set size usage (VmHWM): 3052.19 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357_reco_stage1_20250908T154413_keepup.root
\tHists: np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357_reco_stage1_20250908T154413_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357_reco_stage1_20250908T154413_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357_reco_stage1_20250908T154413_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346254.0,
"core.end_time": 1757346254.0,
"core.events": [
7958,
7978,
7998,
8018,
8038,
8058,
8078,
8098,
8118,
8138,
8158,
8178,
8198,
8218,
8238,
8258,
8278,
8298,
8318,
8338,
8358,
8378,
8398,
8418,
8438,
8458,
8478
],
"core.event_count": 27,
"core.first_event_number": 7958,
"core.last_event_number": 8478,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39343
],
"core.runs_subruns": [
3934300001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357_reco_stage1_20250908T154413_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346254.0,
"core.end_time": 1757346254.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39343
],
"core.runs_subruns": [
3934300001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039343_0015_df-s05-d4_dw_0_20250908T123357.hdf5"
}
]
} 