-
Notifications
You must be signed in to change notification settings - Fork 42
some higher volume mods for the vps #765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
tavdog
wants to merge
15
commits into
main
Choose a base branch
from
dev
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
aa15b28
semaphore to enforce max 5 concurrent renders
tavdog 18bad88
add render metrics
tavdog 869108f
env var for maxconcurrentrenders
tavdog 1dbc50d
print 10 sec stats to log.
tavdog 394240f
modify 10 sec stats
tavdog 5674217
add admin stats dashboard
tavdog 8cb8288
auto refresh the admin dashboard
tavdog c652fb4
skip render if no slots open
tavdog cbb6a8d
print load avg in stats line
tavdog f24a782
tyrp
tavdog ee1889c
remove periodic render stats
tavdog 71d3ffd
semaphors for /next
tavdog c864b68
use atomic.Int64
tavdog 4cd7681
Apply suggestions from code review
tavdog 5bb4f4d
fix rendersem
tavdog File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,6 +15,17 @@ import ( | |
| func (s *Server) handleNextApp(w http.ResponseWriter, r *http.Request) { | ||
| id := r.PathValue("id") | ||
|
|
||
| renderMetrics.RecordRequest() | ||
|
|
||
| // Acquire per-device semaphore to prevent queue backup from same device. | ||
| // If already processing a request for this device, return cached image immediately. | ||
| if !s.acquireDeviceSemaphore(id) { | ||
| slog.Debug("Device busy, serving cached image", "device", id) | ||
| s.serveCachedImageForDevice(w, r, id) | ||
| return | ||
| } | ||
| defer s.releaseDeviceSemaphore(id) | ||
|
|
||
| var device *data.Device | ||
| if d, err := DeviceFromContext(r.Context()); err == nil { | ||
| device = d | ||
|
|
@@ -107,9 +118,13 @@ func (s *Server) handleNextApp(w http.ResponseWriter, r *http.Request) { | |
| // Send default image if error (or not found) | ||
| slog.Error("Failed to get next app image", "device", device.ID, "error", err) | ||
| s.sendDefaultImage(w, r, device) | ||
| webpMetrics.RecordWebPServed(0) | ||
| webpMetrics.RecordUniqueDevice(device.ID) | ||
|
Comment on lines
+121
to
+122
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| return | ||
| } | ||
|
|
||
| webpMetrics.RecordUniqueDevice(device.ID) | ||
|
|
||
| // For HTTP devices, we assume "Sent" equals "Displaying" (or roughly so). | ||
| // We update DisplayingApp here so the Preview uses the explicit field instead of fallback. | ||
| if app != nil { | ||
|
|
@@ -142,6 +157,8 @@ func (s *Server) handleNextApp(w http.ResponseWriter, r *http.Request) { | |
| dwell := device.GetEffectiveDwellTime(app) | ||
| w.Header().Set("Tronbyt-Dwell-Secs", fmt.Sprintf("%d", dwell)) | ||
|
|
||
| webpMetrics.RecordWebPServed(len(imgData)) | ||
|
|
||
| if _, err := w.Write(imgData); err != nil { | ||
| slog.Error("Failed to write image data to response", "error", err) | ||
| // Log error, but can't change HTTP status after writing headers. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,267 @@ | ||
| package server | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "log/slog" | ||
| "os" | ||
| "strconv" | ||
| "strings" | ||
| "sync" | ||
| "sync/atomic" | ||
| "time" | ||
| ) | ||
|
|
||
| type RenderMetrics struct { | ||
| activeCount atomic.Int64 | ||
| queuedCount atomic.Int64 | ||
| totalCount atomic.Int64 | ||
| failedCount atomic.Int64 | ||
| totalDur int64 // nanoseconds | ||
| maxDur atomic.Int64 | ||
|
|
||
| // Sliding window tracking (timestamps of events in last 60 seconds) | ||
| mu sync.Mutex | ||
| rendersByMinute []int64 // timestamps of renders | ||
| reqsByMinute []int64 // timestamps of requests | ||
| } | ||
|
|
||
| var renderMetrics RenderMetrics | ||
|
|
||
| type WebPMetrics struct { | ||
| servedCount atomic.Int64 | ||
| renderCount atomic.Int64 | ||
| bytesServed atomic.Int64 | ||
| uniqueMu sync.Mutex | ||
| uniqueDevices map[string]int64 // device ID -> last seen timestamp | ||
|
|
||
| // Sliding window tracking | ||
| mu sync.Mutex | ||
| webpsByMinute []int64 // timestamps of webp serves | ||
| } | ||
|
|
||
| var webpMetrics WebPMetrics | ||
|
|
||
| const windowDuration = 60 * time.Second | ||
|
|
||
| func (m *RenderMetrics) StartRender() { | ||
| m.activeCount.Add(1) | ||
| m.queuedCount.Add(1) | ||
| } | ||
|
|
||
| func (m *RenderMetrics) EndRender(dur time.Duration, failed bool) { | ||
| m.activeCount.Add(-1) | ||
| m.queuedCount.Add(-1) | ||
| m.totalCount.Add(1) | ||
| atomic.AddInt64(&m.totalDur, int64(dur)) | ||
|
|
||
| currentMax := m.maxDur.Load() | ||
| if int64(dur) > currentMax { | ||
| m.maxDur.Store(int64(dur)) | ||
| } | ||
|
|
||
| if failed { | ||
| m.failedCount.Add(1) | ||
| } | ||
|
|
||
| now := time.Now().Unix() | ||
| m.mu.Lock() | ||
| m.rendersByMinute = append(m.rendersByMinute, now) | ||
| m.mu.Unlock() | ||
| } | ||
|
|
||
| func (m *RenderMetrics) RecordRequest() { | ||
| now := time.Now().Unix() | ||
| m.mu.Lock() | ||
| m.reqsByMinute = append(m.reqsByMinute, now) | ||
| m.mu.Unlock() | ||
| } | ||
|
|
||
| func (m *RenderMetrics) ActiveCount() int64 { | ||
| return m.activeCount.Load() | ||
| } | ||
|
|
||
| func (m *RenderMetrics) AvgDuration() time.Duration { | ||
| total := m.totalCount.Load() | ||
| if total == 0 { | ||
| return 0 | ||
| } | ||
| return time.Duration(m.totalDur / total) | ||
| } | ||
|
|
||
| func (m *RenderMetrics) MaxDuration() time.Duration { | ||
| return time.Duration(m.maxDur.Load()) | ||
| } | ||
|
|
||
| func (m *RenderMetrics) TotalCount() int64 { | ||
| return m.totalCount.Load() | ||
| } | ||
|
|
||
| func (m *RenderMetrics) FailedCount() int64 { | ||
| return m.failedCount.Load() | ||
| } | ||
|
|
||
| func (m *RenderMetrics) QueuedCount() int64 { | ||
| return m.queuedCount.Load() | ||
| } | ||
|
|
||
| func (m *RenderMetrics) RendersPerMin() int64 { | ||
| m.mu.Lock() | ||
| defer m.mu.Unlock() | ||
| cutoff := time.Now().Add(-windowDuration).Unix() | ||
| var count int64 | ||
| for _, t := range m.rendersByMinute { | ||
| if t >= cutoff { | ||
| count++ | ||
| } | ||
| } | ||
| return count | ||
| } | ||
|
|
||
| func (m *RenderMetrics) ReqsPerMin() int64 { | ||
| m.mu.Lock() | ||
| defer m.mu.Unlock() | ||
| cutoff := time.Now().Add(-windowDuration).Unix() | ||
| var count int64 | ||
| for _, t := range m.reqsByMinute { | ||
| if t >= cutoff { | ||
| count++ | ||
| } | ||
| } | ||
| return count | ||
| } | ||
|
|
||
| func (w *WebPMetrics) RecordWebPServed(bytes int) { | ||
| w.servedCount.Add(1) | ||
| w.bytesServed.Add(int64(bytes)) | ||
|
|
||
| now := time.Now().Unix() | ||
| w.mu.Lock() | ||
| w.webpsByMinute = append(w.webpsByMinute, now) | ||
| w.mu.Unlock() | ||
| } | ||
|
|
||
| func (w *WebPMetrics) RecordRender() { | ||
| w.renderCount.Add(1) | ||
| } | ||
|
|
||
| func (w *WebPMetrics) RecordUniqueDevice(deviceID string) { | ||
| now := time.Now().Unix() | ||
| w.uniqueMu.Lock() | ||
| if w.uniqueDevices == nil { | ||
| w.uniqueDevices = make(map[string]int64) | ||
| } | ||
| w.uniqueDevices[deviceID] = now | ||
| w.uniqueMu.Unlock() | ||
| } | ||
|
|
||
| func (w *WebPMetrics) LogStats() { | ||
| served := w.servedCount.Swap(0) | ||
| renders := w.renderCount.Swap(0) | ||
|
|
||
| cutoff := time.Now().Add(-windowDuration).Unix() | ||
| w.uniqueMu.Lock() | ||
| var uniqueDevs int64 | ||
| for _, lastSeen := range w.uniqueDevices { | ||
| if lastSeen >= cutoff { | ||
| uniqueDevs++ | ||
| } | ||
| } | ||
| // Clean up old entries | ||
| for id, lastSeen := range w.uniqueDevices { | ||
| if lastSeen < cutoff { | ||
| delete(w.uniqueDevices, id) | ||
| } | ||
| } | ||
| w.uniqueMu.Unlock() | ||
|
|
||
| loadAvg1m := getLoadAverage() | ||
| if served > 0 { | ||
| slog.Info(fmt.Sprintf("Stats ------ : %.1f - %d / %d ", loadAvg1m, served, renders)) | ||
|
Comment on lines
+177
to
+179
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| } | ||
| } | ||
|
|
||
| func getLoadAverage() float64 { | ||
| data, err := os.ReadFile("/proc/loadavg") | ||
| if err != nil { | ||
| return 0 | ||
| } | ||
| parts := strings.Split(string(data), " ") | ||
| if len(parts) < 1 { | ||
| return 0 | ||
| } | ||
| f, err := strconv.ParseFloat(parts[0], 64) | ||
| if err != nil { | ||
| return 0 | ||
| } | ||
| return f | ||
| } | ||
|
|
||
| func (w *WebPMetrics) ServedCount() int64 { | ||
| return w.servedCount.Load() | ||
| } | ||
|
|
||
| func (w *WebPMetrics) RenderCount() int64 { | ||
| return w.renderCount.Load() | ||
| } | ||
|
|
||
| func (w *WebPMetrics) BytesServed() int64 { | ||
| return w.bytesServed.Load() | ||
| } | ||
|
|
||
| func (w *WebPMetrics) WebpsPerMin() int64 { | ||
| w.mu.Lock() | ||
| defer w.mu.Unlock() | ||
| cutoff := time.Now().Add(-windowDuration).Unix() | ||
| var count int64 | ||
| for _, t := range w.webpsByMinute { | ||
| if t >= cutoff { | ||
| count++ | ||
| } | ||
| } | ||
| return count | ||
| } | ||
|
|
||
| func (w *WebPMetrics) UniqueDevicesPerMin() int64 { | ||
| cutoff := time.Now().Add(-windowDuration).Unix() | ||
| w.uniqueMu.Lock() | ||
| defer w.uniqueMu.Unlock() | ||
| var count int64 | ||
| for _, lastSeen := range w.uniqueDevices { | ||
| if lastSeen >= cutoff { | ||
| count++ | ||
| } | ||
| } | ||
| return count | ||
| } | ||
|
|
||
| type StatsSnapshot struct { | ||
| ActiveRenders int64 | ||
| QueuedRenders int64 | ||
| TotalRenders int64 | ||
| FailedRenders int64 | ||
| AvgRenderMs int64 | ||
| MaxRenderMs int64 | ||
| RendersPerMin int64 | ||
| ReqsPerMin int64 | ||
| WebpsServed int64 | ||
| WebpsPerMin int64 | ||
| BytesServedMB float64 | ||
| UniqueDevsPerMin int64 | ||
| } | ||
|
|
||
| func GetStatsSnapshot() StatsSnapshot { | ||
| return StatsSnapshot{ | ||
| ActiveRenders: renderMetrics.ActiveCount(), | ||
| QueuedRenders: renderMetrics.QueuedCount(), | ||
| TotalRenders: renderMetrics.TotalCount(), | ||
| FailedRenders: renderMetrics.FailedCount(), | ||
| AvgRenderMs: renderMetrics.AvgDuration().Milliseconds(), | ||
| MaxRenderMs: renderMetrics.MaxDuration().Milliseconds(), | ||
| RendersPerMin: renderMetrics.RendersPerMin(), | ||
| ReqsPerMin: renderMetrics.ReqsPerMin(), | ||
| WebpsServed: webpMetrics.ServedCount(), | ||
| WebpsPerMin: webpMetrics.WebpsPerMin(), | ||
| BytesServedMB: float64(webpMetrics.BytesServed()) / (1024 * 1024), | ||
| UniqueDevsPerMin: webpMetrics.UniqueDevicesPerMin(), | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's good to provide a default value for
MaxConcurrentRenders, but consider adding a comment explaining why5was chosen as the default. Is it based on testing or hardware constraints? This will help future maintainers understand the rationale behind the default value.Also, consider adding a validation to ensure that the value is not negative, and log a warning if it is.