Jobsub ID 36279.132@dunegpschedd02.fnal.gov
| Jobsub ID | 36279.132@dunegpschedd02.fnal.gov |
| Workflow ID | 2329 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 10:27:50 |
| Site | UK_Glasgow |
| Entry | CLAS12_T3_UK_ScotGrid_GLA_ce04_scitok |
| Last heartbeat | 2025-09-08 15:52:48 |
| From worker node | Hostname | wn-d20-031.beowulf.cluster |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 12:47:23 |
| Input files | vd-protodune:np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 3h (10828s) |
| CPU time | 1h (3925s = 36%) |
| Max RSS bytes | 2834706432 (2703 MiB) |
| Outputting started | 2025-09-08 15:47:52 |
| Output files | |
| Finished | 2025-09-08 15:52:48 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
at call=53 with 8
[16:42:28.233] D [ glue ] frame sink sees EOS
[16:42:28.233] D [ pgraph ] <Pgrapher:> graph execution complete
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.07 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.39 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.37 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.63 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.44 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.27 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.18 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.61 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Aux::Resampler : 0.46 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Aux::Resampler : 0.43 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Aux::Resampler : 0.38 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Aux::Resampler : 0.36 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:42:28.233] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:42:28.233] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:42:28.234] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:42:28.234] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:42:28.234] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:42:28.234] I [ timer ] Timer: Total node execution : 58.70000006072223 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 43065 traces tagged "gauss"
FrameSaver: q=7.64994e+06 n=895779 tag=gauss
wclsFrameSaver: saving 54373 traces tagged "wiener"
FrameSaver: q=8.18436e+06 n=847289 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 344 XUs and 388 XVs -> 24 XUVs
C:0 T:1 928 XUs and 949 XVs -> 38 XUVs
C:0 T:2 105 XUs and 62 XVs -> 4 XUVs
C:0 T:3 1010 XUs and 662 XVs -> 55 XUVs
C:0 T:4 784 XUs and 771 XVs -> 33 XUVs
C:0 T:5 2421 XUs and 2326 XVs -> 95 XUVs
C:0 T:6 594 XUs and 712 XVs -> 18 XUVs
C:0 T:7 2461 XUs and 1709 XVs -> 104 XUVs
C:0 T:8 10406 XUs and 16842 XVs -> 2821 XUVs
C:0 T:9 8302 XUs and 11325 XVs -> 1072 XUVs
C:0 T:10 919 XUs and 1427 XVs -> 270 XUVs
C:0 T:11 2325 XUs and 3721 XVs -> 195 XUVs
C:0 T:12 4327 XUs and 5717 XVs -> 378 XUVs
C:0 T:13 641 XUs and 1291 XVs -> 57 XUVs
C:0 T:14 10903 XUs and 18279 XVs -> 2861 XUVs
C:0 T:15 13148 XUs and 6638 XVs -> 1290 XUVs
9315 XUVs total
1851 collection wire objects
9315 potential space points
Neighbour search...
1082817 tests to find 479066 neighbours
Iterating with no regularization...
Begin: 7.50111e+08
0 6.88906e+08
1 6.84511e+08
2 6.84016e+08
Now with regularization...
Begin: 6.74272e+08
0 6.74027e+08
08-Sep-2025 16:43:40 BST Closed output file "np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240_reco_stage1_20250908T154340_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 96.0805 388.716 6143.41 159.105 1129.35 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.1222e-05 0.000133908 0.000329081 0.000117461 5.10567e-05 27
produce:tpcrawdecoder:PDVDTPCReader 16.8738 239.043 5907.18 20.3044 1111.62 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.056558 0.0583699 0.0700562 0.0569764 0.00396329 27
produce:pdvddaphne:DAPHNEReaderPDVD 3.16576 3.92361 4.96605 3.90032 0.459938 27
produce:ophit:OpHitFinder 0.0351027 0.0513588 0.0591506 0.0515525 0.00459115 27
produce:opflash:OpFlashFinderVerticalDrift 0.00664511 0.0162783 0.0246831 0.0160262 0.00426068 27
produce:wclsdatavd:WireCellToolkit 50.4299 59.9974 99.2611 58.0546 8.69136 27
produce:gaushit:GausHitFinder 0.570336 1.3109 2.10243 1.32216 0.400905 27
produce:nhitsfilter:NumberOfHitsFilter 0.00027043 0.000549716 0.00114145 0.000507839 0.000177858 27
produce:reco3d:SpacePointSolver 5.11479 12.8105 23.9378 12.864 3.91321 27
produce:hitpdune:DisambigFromSpacePoints 0.076128 0.225329 0.492525 0.212625 0.094966 27
produce:pandora:StandardPandora 14.7342 65.4896 163.425 52.284 37.2511 27
produce:pandoraTrack:LArPandoraTrackCreation 0.239451 1.06869 2.97048 0.925504 0.611883 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0200362 0.0349359 0.0642997 0.0344485 0.0102659 27
[art]:TriggerResults:TriggerResultInserter 2.3494e-05 5.60825e-05 0.000157758 5.1277e-05 2.74534e-05 27
end_path:out1:RootOutput 4.128e-06 9.94481e-06 4.4404e-05 8.737e-06 7.57176e-06 27
end_path:out1:RootOutput(write) 3.76949 4.59954 7.85851 4.46732 0.766742 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4829.85 MB
Peak resident set size usage (VmHWM): 2834.71 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240_reco_stage1_20250908T154340_keepup.root
\tHists: np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240_reco_stage1_20250908T154340_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240_reco_stage1_20250908T154340_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240_reco_stage1_20250908T154340_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346221.0,
"core.end_time": 1757346221.0,
"core.events": [
1443601,
1443621,
1443641,
1443661,
1443681,
1443701,
1443721,
1443741,
1443761,
1443781,
1443801,
1443821,
1443841,
1443861,
1443881,
1443901,
1443921,
1443941,
1443961,
1443981,
1444001,
1444021,
1444041,
1444061,
1444081,
1444101,
1444121
],
"core.event_count": 27,
"core.first_event_number": 1443601,
"core.last_event_number": 1444121,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240_reco_stage1_20250908T154340_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346221.0,
"core.end_time": 1757346221.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2753_df-s01-d3_dw_0_20250908T014240.hdf5"
}
]
} 