Jobsub ID 36338.32@dunegpschedd02.fnal.gov
| Jobsub ID | 36338.32@dunegpschedd02.fnal.gov |
| Workflow ID | 2323 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:55:52 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2025-09-08 15:52:42 |
| From worker node | Hostname | cn064.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:10:24 |
| Input files | vd-protodune:np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5863s) |
| CPU time | 1h (5280s = 90%) |
| Max RSS bytes | 3083800576 (2940 MiB) |
| Outputting started | 2025-09-08 15:48:08 |
| Output files | |
| Finished | 2025-09-08 15:52:42 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
frame sink sees EOS
[16:43:01.906] D [ pgraph ] <Pgrapher:> graph execution complete
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.85 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.2 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.07 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.56 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.79 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.27 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.09 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.31 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Aux::Resampler : 0.52 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Aux::Resampler : 0.32 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Aux::Resampler : 0.31 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Aux::Resampler : 0.3 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:01.906] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:01.906] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:43:01.906] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:43:01.906] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:43:01.906] I [ timer ] Timer: Total node execution : 66.68000032939017 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 36225 traces tagged "gauss"
FrameSaver: q=6.79142e+06 n=793270 tag=gauss
wclsFrameSaver: saving 45081 traces tagged "wiener"
FrameSaver: q=7.33722e+06 n=758798 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11 XUs and 14 XVs -> 1 XUVs
C:0 T:1 101 XUs and 104 XVs -> 6 XUVs
C:0 T:2 67 XUs and 160 XVs -> 0 XUVs
C:0 T:3 481 XUs and 484 XVs -> 6 XUVs
C:0 T:4 125 XUs and 152 XVs -> 7 XUVs
C:0 T:5 367 XUs and 449 XVs -> 13 XUVs
C:0 T:6 4394 XUs and 4145 XVs -> 358 XUVs
C:0 T:7 254 XUs and 345 XVs -> 9 XUVs
C:0 T:8 8083 XUs and 12753 XVs -> 2066 XUVs
C:0 T:9 7113 XUs and 16949 XVs -> 2045 XUVs
C:0 T:10 3307 XUs and 5168 XVs -> 1080 XUVs
C:0 T:11 1399 XUs and 2205 XVs -> 78 XUVs
C:0 T:12 2132 XUs and 2817 XVs -> 264 XUVs
C:0 T:13 2664 XUs and 4876 XVs -> 593 XUVs
C:0 T:14 7646 XUs and 28791 XVs -> 2618 XUVs
C:0 T:15 9721 XUs and 11151 XVs -> 2109 XUVs
11253 XUVs total
1687 collection wire objects
11253 potential space points
Neighbour search...
1288339 tests to find 574620 neighbours
Iterating with no regularization...
Begin: 4.64115e+08
0 4.10328e+08
1 4.06509e+08
2 4.05801e+08
3 4.05563e+08
Now with regularization...
Begin: 3.91824e+08
0 3.91593e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 16:44:14 BST Closed output file "np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250_reco_stage1_20250908T154414_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 106.851 205.579 372.134 185.76 66.4488 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.3428e-05 0.000127255 0.000292809 0.00011377 4.67679e-05 27
produce:tpcrawdecoder:PDVDTPCReader 12.0869 14.8664 34.7822 13.034 5.56929 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0330009 0.0351618 0.0453267 0.033358 0.0042004 27
produce:pdvddaphne:DAPHNEReaderPDVD 4.71087 6.4374 8.45128 6.18055 1.17849 27
produce:ophit:OpHitFinder 0.0377879 0.0611346 0.0914417 0.051131 0.0170587 27
produce:opflash:OpFlashFinderVerticalDrift 0.0074494 0.017724 0.0341475 0.015156 0.00723138 27
produce:wclsdatavd:WireCellToolkit 60.8554 79.0986 103.525 75.3061 13.2537 27
produce:gaushit:GausHitFinder 0.733477 1.78369 2.88158 1.71014 0.584043 27
produce:nhitsfilter:NumberOfHitsFilter 0.000311838 0.000621739 0.00150341 0.000617501 0.00022321 27
produce:reco3d:SpacePointSolver 5.34112 13.3488 21.5576 13.1495 4.40399 27
produce:hitpdune:DisambigFromSpacePoints 0.0908881 0.301414 0.535591 0.284211 0.127068 27
produce:pandora:StandardPandora 16.3876 82.7147 236.825 72.0939 52.4948 27
produce:pandoraTrack:LArPandoraTrackCreation 0.405767 2.0612 6.1413 1.8477 1.28698 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0145324 0.0376356 0.0704695 0.0350115 0.0145505 27
[art]:TriggerResults:TriggerResultInserter 3.2891e-05 5.64317e-05 0.000159158 5.5769e-05 2.45711e-05 27
end_path:out1:RootOutput 3.931e-06 1.16396e-05 5.0613e-05 9.717e-06 8.43747e-06 27
end_path:out1:RootOutput(write) 3.16074 4.75255 6.7473 4.90838 1.1345 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4971.06 MB
Peak resident set size usage (VmHWM): 3083.8 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250_reco_stage1_20250908T154414_keepup.root
\tHists: np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250_reco_stage1_20250908T154421_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250_reco_stage1_20250908T154414_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250_reco_stage1_20250908T154414_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346261.0,
"core.end_time": 1757346261.0,
"core.events": [
884465,
884485,
884505,
884525,
884545,
884565,
884585,
884605,
884625,
884645,
884665,
884685,
884705,
884725,
884745,
884765,
884785,
884805,
884825,
884845,
884865,
884885,
884905,
884925,
884945,
884965,
884985
],
"core.event_count": 27,
"core.first_event_number": 884465,
"core.last_event_number": 884985,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250_reco_stage1_20250908T154421_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346261.0,
"core.end_time": 1757346261.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1689_df-s02-d3_dw_0_20250907T035250.hdf5"
}
]
} 