aom: rtc: Fix to keyframe coding for speed 11 video

From 4b7e5b33251220107ac60aba5f3781c4f69960ad Mon Sep 17 00:00:00 2001
From: Marco Paniconi <[EMAIL REDACTED]>
Date: Wed, 6 Mar 2024 11:54:10 -0800
Subject: [PATCH] rtc: Fix to keyframe coding for speed 11 video

For speed 11 video, <= VGA: the existing key frame
coding was using hybrid (rd/nonrd) coding for QVGA,
and for VGA is was using fixed block partition, which
was not intended (bug).

This CL fixes this to use nonrd pickmode for keyframe
coding (set hybird_intra_pickmode = 0), which has some
reasonable quality/speed tradeoff than the existing setting,
and avoids the use of fixed partition for vga (where
use_fast_fixed_part = 1).

stats change (avg/ovr/ssim, IC speedup):
rtc_derf: 0.3/0.3/0.2 bdrate, ~1.2% IC speedup
rtc (vga only): -0.4/-0.9/-2 bdrate, ~0.3 IC slowdown

Change-Id: Idc0c40b19a019d6e62e22cbde4ab412d509adb36
---
 av1/encoder/encodeframe.c    | 3 ++-
 av1/encoder/speed_features.c | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/av1/encoder/encodeframe.c b/av1/encoder/encodeframe.c
index 784840959..a9214f77c 100644
--- a/av1/encoder/encodeframe.c
+++ b/av1/encoder/encodeframe.c
@@ -536,7 +536,8 @@ static AOM_INLINE void encode_nonrd_sb(AV1_COMP *cpi, ThreadData *td,
 #endif
   // Set the partition
   if (sf->part_sf.partition_search_type == FIXED_PARTITION || seg_skip ||
-      (sf->rt_sf.use_fast_fixed_part && x->sb_force_fixed_part == 1)) {
+      (sf->rt_sf.use_fast_fixed_part && x->sb_force_fixed_part == 1 &&
+       !frame_is_intra_only(cm))) {
     // set a fixed-size partition
     av1_set_offsets(cpi, tile_info, x, mi_row, mi_col, sb_size);
     BLOCK_SIZE bsize_select = sf->part_sf.fixed_partition_size;
diff --git a/av1/encoder/speed_features.c b/av1/encoder/speed_features.c
index 0c1447b7b..115ce06d9 100644
--- a/av1/encoder/speed_features.c
+++ b/av1/encoder/speed_features.c
@@ -1476,6 +1476,7 @@ static void set_rt_speed_feature_framesize_dependent(const AV1_COMP *const cpi,
     for (int i = 0; i < BLOCK_SIZES; ++i) {
       sf->rt_sf.intra_y_mode_bsize_mask_nrd[i] = INTRA_DC;
     }
+    sf->rt_sf.hybrid_intra_pickmode = 0;
   }
   // Setting for SVC, or when the ref_frame_config control is
   // used to set the reference structure.