1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
|
# FUSE VFS Performance Handoff (SkyrimSE / MO2 Linux Port)
## Context
- Project: MO2 Linux Port with C
- Test instance: `/home/luke/Games/ElderTeej`
- Symptom: VFS tree build is very fast (`~300ms`), but Skyrim startup can stall for many minutes.
- Baseline expectation from user: startup should be around `2-3 minutes`.
## Core Diagnosis
The slow path is **not** initial VFS construction.
The bottleneck is sustained runtime metadata traffic (Wine/game probing), especially:
- `getattr`
- `readdir`
- (previously) `ioctl` probes
Observed pattern during bad runs:
- Very high `getattr/readdir/ioctl`
- Very low `read`
- Process appears mostly idle in btop (low MB read/write), yet VFS op counters climb rapidly
## Why This Happens
FUSE builds a tree once, but then serves kernel/user-space requests continuously.
Games + Wine often do heavy existence/stat/directory probing before and during menu load.
So startup cost can be dominated by metadata round-trips even if actual file reads are small.
## Major Evidence from Logs
- Early problematic runs showed huge ioctl growth (millions) with low reads.
- After fixes, `ioctl` can be driven to `0`.
- Even with `ioctl=0`, `readdir/getattr` remained very high in some phases:
- root (`""`)
- `ShaderCache/*`
- `meshes/actors/character`
- `meshes/actors/character/facegenmorphs`
- `Seq`
## Implemented Changes (Chronological Summary)
### 1) Persistent file handles for open files (big fix)
- Reworked open/read/write/release to use handle state.
- Avoided reopening backing files on every read.
- `write` path moved to `pwrite`.
- `release` now closes tracked FDs.
Impact:
- Reduced repeated open/close overhead for active handles.
---
### 2) Added ioctl handler (then corrected behavior)
- Implemented `mo2_ioctl`.
- Returned `ENOTTY` for known probe-heavy commands (`VFAT_IOCTL_READDIR_BOTH`, `FS_IOC_GETFLAGS`).
- Added ioctl hotloop diagnostics.
Impact:
- Helped identify ioctl storms.
---
### 3) Enabled directory operation caching primitives
- Added `opendir` / `releasedir`.
- Added directory handle cache for entries.
- Added `readdirplus` callback and capability flags initially.
- Added negative lookup caching (`fuse_reply_entry` with `ino=0`, TTL).
- Increased entry/attr TTL values in replies.
Impact:
- Reduced some repeated rebuild work, but did not fully eliminate metadata storm.
---
### 4) Added instrumentation for hotspot visibility
- Global op counters (`lookup/getattr/readdir/open/read/ioctl`).
- Top hot paths logging for lookup hit/miss/getattr/readdir.
- Sampling mechanism to keep logs manageable.
Impact:
- Confirmed dominant hotspots and phase shifts.
---
### 5) Stale mount cleanup hardening
- Added proactive cleanup for stale `mo2linux` mounts (including trash paths).
Impact:
- Stability/cleanup improvement, not direct startup speedup.
---
### 6) Stabilized synthetic directory timestamps
- Virtual dir stat times changed from `now` per request to fixed constant.
Impact:
- Prevents needless cache invalidation behavior due to changing timestamps.
---
### 7) Lock/contention reduction in inode path
- Switched inode mutex to shared mutex.
- Added inode fast lookup (`get`) separate from `getOrCreate`.
- Reduced exclusive lock pressure on read-heavy paths.
Impact:
- Better concurrency under FUSE multithread loop.
---
### 8) Per-handle path retention in readdir path
- Stored directory path on open dir handles to avoid repeated inode-to-path resolution.
Impact:
- Reduced overhead in hot `readdir` loop.
---
### 9) Prebuilt encoded directory blobs
- Built and reused encoded `readdir` and `readdirplus` blobs.
- Added global per-path blob caches.
Impact:
- Makes repeated directory listing replies cheaper.
---
### 10) FD exhaustion fix (critical stability fix)
- Runtime crash seen:
- `GLib-ERROR: Creating pipes for GWakeup: Too many open files`
- Cause: too many retained open handles.
- Fix:
- Do **not** keep persistent FD for read-only handles.
- Open temp FD in read path when needed, close immediately.
- Keep persistent FD only for writable handles.
- Added `open_handles` counter in log line.
Impact:
- Stabilized handle growth behavior and avoided EMFILE crash path.
---
### 11) Disabled custom ioctl callback in ops table
- Stopped wiring `ops->ioctl`.
Impact:
- `ioctl` count dropped to `0` in user logs.
- Removed large class of user-space ioctl churn.
---
### 12) Removed `default_permissions` mount option (experiment)
- Tested to reduce permission-check-driven metadata round-trips.
Impact:
- Mixed/limited; metadata storm still exists in hot phases.
---
### 13) Disabled `readdirplus` capability/callback (experiment)
- Stopped advertising/using readdirplus.
Impact:
- Profile changed; in latest successful run startup reached `2m40s`.
---
## Current State (Most Important)
- User reported successful startup in **~2 minutes 40 seconds**.
- `ioctl=0`.
- `open_handles` still can grow into the thousands in long runs, but EMFILE crash path improved substantially after RO FD change.
- Remaining hot metadata phases are path-specific (shader cache and facegen/character meshes).
## Remaining Known Issues
- Menu/runtime phases still generate heavy `getattr/readdir` on certain directories.
- `open_handles` can still increase significantly depending on workload phase.
## What Appears to Have Helped Most
1. Removing ioctl churn (`ioctl=0`).
2. Avoiding persistent RO file FDs (stability fix).
3. Directory response caching plus reduced overhead paths.
4. Readdirplus/permissions experiments likely affected metadata profile depending on phase.
## Suggested Next Experiments (if further optimization is desired)
1. Add small in-process `getattr` LRU cache (short TTL, e.g. 0.5-1.0s) keyed by inode/path.
2. Introduce targeted caches for hottest directories:
- `meshes/actors/character`
- `meshes/actors/character/facegenmorphs`
- `ShaderCache/*`
- `Seq`
3. Add optional “performance mode” config flag:
- less verbose logging
- larger TTLs
- aggressive directory blob reuse
4. Track `release`/`releasedir` deltas in logs to verify handle churn balance.
5. Validate `ulimit -n` at launch and warn if too low.
## Useful Log Fields for Future Triage
- `ops lookup=... getattr=... readdir=... open=... read=... ioctl=... open_handles=...`
- `hot readdir: ...`
- `hot getattr: ...`
- `hot lookup_hit/lookup_miss: ...`
## Bottom Line
The root misunderstanding was “fast VFS build should imply fast startup.”
In practice, startup was dominated by post-build metadata probing.
After multiple fixes, the system reached user-acceptable startup (`~2m40s`) with much better stability and no ioctl storm.
|
