libtiff: incorporate a fix for possible fax3 decoding overflow (CVE-2011-0192)

https://github.com/libsdl-org/libtiff/commit/940d1ec552d3563941f8430e559c4cebd9d1c3a8

From 940d1ec552d3563941f8430e559c4cebd9d1c3a8 Mon Sep 17 00:00:00 2001
From: Frank Warmerdam <[EMAIL REDACTED]>
Date: Thu, 3 Mar 2011 01:36:10 +0000
Subject: [PATCH] incorporate a fix for possible fax3 decoding overflow
 (CVE-2011-0192)

---
 ChangeLog          | 11 +++++++++++
 libtiff/tif_fax3.h |  6 +++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index 6f1e8ec9..4ed5e0e5 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,14 @@
+2011-03-02  Frank Warmerdam  <warmerdam@pobox.com>
+
+	* libtiff/tif_fax3.h: Protect against a fax VL(n) codeword commanding 
+	a move left.  Without this, a malicious input file can generate an 
+	indefinitely large series of runs without a0 ever reaching the right 
+	margin, thus overrunning our buffer of run lengths.  Per CVE-2011-0192.
+	This is a modified version of a patch proposed by Drew Yao of Apple 
+	Product Security.  It adds an unexpected() report, and disallows the 
+	equality case, since emitting a run without increasing a0 still allows 
+	buffer overrun.
+
 2011-02-25  Andrey Kiselev  <dron@ak4719.spb.edu>
 
 	* libtiff/tif_dirwrite.c: Avoid undefined behaviour when casting from
diff --git a/libtiff/tif_fax3.h b/libtiff/tif_fax3.h
index 40718bcf..f4fc8c8d 100644
--- a/libtiff/tif_fax3.h
+++ b/libtiff/tif_fax3.h
@@ -1,4 +1,4 @@
-/* $Id: tif_fax3.h,v 1.5.2.1 2010-06-08 18:50:42 bfriesen Exp $ */
+/* $Id: tif_fax3.h,v 1.5.2.2 2011-03-03 01:36:10 fwarmerdam Exp $ */
 
 /*
  * Copyright (c) 1990-1997 Sam Leffler
@@ -478,6 +478,10 @@ done1d:									\
 	    break;							\
 	case S_VL:							\
 	    CHECK_b1;							\
+	    if (b1 <= (int) (a0 + TabEnt->Param)) {			\
+		unexpected("VL", a0);					\
+		goto eol2d;						\
+	    }								\
 	    SETVALUE(b1 - a0 - TabEnt->Param);				\
 	    b1 -= *--pb;						\
 	    break;							\